2024-08-21T21:54:04.472 INFO:root:teuthology version: 1.2.1.dev2+g4d4c5e3 2024-08-21T21:54:04.491 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-21T21:54:04.578 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274 branch: wip-yuri7-testing-2024-08-20-1105-quincy description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7866274' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri7-testing-2024-08-20-1105-quincy ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: b5e746614a5ba6992e5be76bf3df447c931f3748 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: b5e746614a5ba6992e5be76bf3df447c931f3748 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py workunit: branch: wip-yuri7-testing-2024-08-20-1105-quincy sha1: b5e746614a5ba6992e5be76bf3df447c931f3748 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 6565 sha1: b5e746614a5ba6992e5be76bf3df447c931f3748 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: wip-yuri7-testing-2024-08-20-1105-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: b5e746614a5ba6992e5be76bf3df447c931f3748 targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMZw3gcuLo5X2pqLvNADs7gKZN4so6LJethSQlhnXNg1d9qR1MWdWYmyGuH5Ktr0PrbjgDDhWg5WK2aczrvW/zA= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBwiuhM1fPYG8eHoRh/WKgq0Bvla0NGDeBYIL52Cb/n8CRt+ckwfBetifQpkEOPmoN0phJKjpnMu2pcB0AwETrs= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 4d4c5e30374204f3b32f867e61bb4e1a3880f672 timestamp: 2024-08-21_20:32:34 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-21T21:54:04.578 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa; will attempt to use it 2024-08-21T21:54:04.579 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks 2024-08-21T21:54:04.579 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-21T21:54:04.581 INFO:teuthology.task.internal:Checking packages... 2024-08-21T21:54:04.602 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b5e746614a5ba6992e5be76bf3df447c931f3748' 2024-08-21T21:54:04.602 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-21T21:54:04.602 INFO:teuthology.packaging:ref: None 2024-08-21T21:54:04.602 INFO:teuthology.packaging:tag: None 2024-08-21T21:54:04.602 INFO:teuthology.packaging:branch: wip-yuri7-testing-2024-08-20-1105-quincy 2024-08-21T21:54:04.602 INFO:teuthology.packaging:sha1: b5e746614a5ba6992e5be76bf3df447c931f3748 2024-08-21T21:54:04.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri7-testing-2024-08-20-1105-quincy 2024-08-21T21:54:04.750 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1456.gb5e74661 2024-08-21T21:54:04.752 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-21T21:54:04.758 INFO:teuthology.task.internal:no buildpackages task found 2024-08-21T21:54:04.758 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-21T21:54:04.763 INFO:teuthology.task.internal:Saving configuration 2024-08-21T21:54:04.777 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-21T21:54:04.783 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-21T21:54:04.807 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-21 21:48:45.384860', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMZw3gcuLo5X2pqLvNADs7gKZN4so6LJethSQlhnXNg1d9qR1MWdWYmyGuH5Ktr0PrbjgDDhWg5WK2aczrvW/zA='} 2024-08-21T21:54:04.829 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-21 21:48:45.383216', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBwiuhM1fPYG8eHoRh/WKgq0Bvla0NGDeBYIL52Cb/n8CRt+ckwfBetifQpkEOPmoN0phJKjpnMu2pcB0AwETrs='} 2024-08-21T21:54:04.829 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-21T21:54:04.835 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2024-08-21T21:54:04.835 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2024-08-21T21:54:04.835 INFO:teuthology.run_tasks:Running task console_log... 2024-08-21T21:54:04.893 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbfb70e7be0>, signals=[15]) 2024-08-21T21:54:04.893 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-21T21:54:04.900 INFO:teuthology.task.internal:Opening connections... 2024-08-21T21:54:04.900 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T21:54:04.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-21T21:54:04.979 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T21:54:04.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-21T21:54:05.053 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-21T21:54:05.060 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-08-21T21:54:05.079 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-08-21T21:54:05.079 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-21T21:54:05.137 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-08-21T21:54:05.138 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-21T21:54:05.138 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-21T21:54:05.138 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-21T21:54:05.138 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-08-21T21:54:05.153 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-08-21T21:54:05.171 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-08-21T21:54:05.171 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-08-21T21:54:05.229 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-21T21:54:05.230 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-21T21:54:05.230 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-21T21:54:05.230 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-08-21T21:54:05.245 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-21T21:54:05.254 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-21T21:54:05.346 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-21T21:54:05.346 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-08-21T21:54:05.349 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-08-21T21:54:05.366 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-21T21:54:05.371 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-21T21:54:05.372 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-08-21T21:54:05.407 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-08-21T21:54:05.442 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-21T21:54:05.498 INFO:teuthology.run_tasks:Running task kernel... 2024-08-21T21:54:05.511 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-21T21:54:05.511 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-21T21:54:05.512 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-21T21:54:05.512 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-21T21:54:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-21T21:54:05.529 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-08-21T21:54:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-21T21:54:05.530 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-08-21T21:54:05.587 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-496.el9.x86_64 2024-08-21T21:54:05.588 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-496.el9.x86_64 2024-08-21T21:54:05.588 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-08-21T21:54:05.589 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-496.el9.x86_64 2024-08-21T21:54:05.589 INFO:teuthology.task.kernel:Running kernel on smithi178: 5.14.0-496.el9.x86_64 2024-08-21T21:54:05.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2024-08-21T21:54:08.401 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.2 MB 00:00 2024-08-21T21:54:09.926 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.2 MB 00:02 2024-08-21T21:54:10.594 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-08-21T21:54:11.768 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-08-21T21:54:15.436 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 6.6 MB 00:01 2024-08-21T21:54:15.894 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.6 MB 00:00 2024-08-21T21:54:16.987 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 18 kB 00:00 2024-08-21T21:54:17.411 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 18 kB 00:00 2024-08-21T21:54:18.249 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 20 MB/s | 22 MB 00:01 2024-08-21T21:54:18.574 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-08-21T21:54:23.579 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-21T21:54:23.888 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-21T21:54:25.513 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-21T21:54:25.513 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-496.el9.x86_64 is already installed. 2024-08-21T21:54:25.598 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:54:25.599 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:54:25.599 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:54:25.710 DEBUG:teuthology.orchestra.run.smithi116:> echo no | sudo yum reinstall kernel || true 2024-08-21T21:54:25.828 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-21T21:54:25.828 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-5.14.0-496.el9.x86_64 is already installed. 2024-08-21T21:54:25.911 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:54:25.912 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:54:25.912 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:54:26.031 DEBUG:teuthology.orchestra.run.smithi178:> echo no | sudo yum reinstall kernel || true 2024-08-21T21:54:26.262 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:03 ago on Wed 21 Aug 2024 09:54:23 PM UTC. 2024-08-21T21:54:26.397 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:54:26.405 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:54:26.405 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-08-21T21:54:26.405 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:54:26.405 INFO:teuthology.orchestra.run.smithi116.stdout:Reinstalling: 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-496.el9 baseos 1.8 M 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 1.8 M 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 0 2024-08-21T21:54:26.406 INFO:teuthology.orchestra.run.smithi116.stderr:Operation aborted. 2024-08-21T21:54:26.459 INFO:teuthology.orchestra.run.smithi116.stdout:Is this ok [y/N]: 2024-08-21T21:54:26.459 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum reinstall -y kernel || true 2024-08-21T21:54:26.632 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:03 ago on Wed 21 Aug 2024 09:54:23 PM UTC. 2024-08-21T21:54:26.770 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:54:26.782 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:54:26.782 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-08-21T21:54:26.782 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:54:26.782 INFO:teuthology.orchestra.run.smithi178.stdout:Reinstalling: 2024-08-21T21:54:26.782 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 5.14.0-496.el9 baseos 1.8 M 2024-08-21T21:54:26.782 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:54:26.783 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:54:26.783 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:54:26.783 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:54:26.783 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.8 M 2024-08-21T21:54:26.783 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 0 2024-08-21T21:54:26.783 INFO:teuthology.orchestra.run.smithi178.stderr:Operation aborted. 2024-08-21T21:54:26.843 INFO:teuthology.orchestra.run.smithi178.stdout:Is this ok [y/N]: 2024-08-21T21:54:26.843 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum reinstall -y kernel || true 2024-08-21T21:54:27.008 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:04 ago on Wed 21 Aug 2024 09:54:23 PM UTC. 2024-08-21T21:54:27.130 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout:Reinstalling: 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-496.el9 baseos 1.8 M 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:54:27.131 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:54:27.132 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:54:27.132 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 1.8 M 2024-08-21T21:54:27.132 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 0 2024-08-21T21:54:27.132 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:54:27.392 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:04 ago on Wed 21 Aug 2024 09:54:23 PM UTC. 2024-08-21T21:54:27.515 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:Reinstalling: 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 5.14.0-496.el9 baseos 1.8 M 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.8 M 2024-08-21T21:54:27.516 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 0 2024-08-21T21:54:27.517 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:54:27.809 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-496.el9.x86_64.rpm 2.7 MB/s | 1.8 MB 00:00 2024-08-21T21:54:27.809 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:54:27.809 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.6 MB/s | 1.8 MB 00:00 2024-08-21T21:54:27.838 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:54:27.873 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-496.el9.x86_64.rpm 7.0 MB/s | 1.8 MB 00:00 2024-08-21T21:54:27.874 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:54:27.874 INFO:teuthology.orchestra.run.smithi178.stdout:Total 5.0 MB/s | 1.8 MB 00:00 2024-08-21T21:54:27.902 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:54:28.111 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:54:28.111 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:54:28.140 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:54:28.141 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:54:28.161 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:54:28.161 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:54:28.189 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:54:28.190 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:54:28.290 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:54:28.366 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:54:28.377 INFO:teuthology.orchestra.run.smithi116.stdout: Reinstalling : kernel-5.14.0-496.el9.x86_64 1/2 2024-08-21T21:54:28.377 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : kernel-5.14.0-496.el9.x86_64 2/2 2024-08-21T21:54:28.434 INFO:teuthology.orchestra.run.smithi178.stdout: Reinstalling : kernel-5.14.0-496.el9.x86_64 1/2 2024-08-21T21:54:28.434 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : kernel-5.14.0-496.el9.x86_64 2/2 2024-08-21T21:54:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-496.el9.x86_64 2/2 2024-08-21T21:54:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-496.el9.x86_64 1/2 2024-08-21T21:54:29.227 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-5.14.0-496.el9.x86_64 2/2 2024-08-21T21:54:29.227 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-5.14.0-496.el9.x86_64 1/2 2024-08-21T21:54:29.909 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-496.el9.x86_64 2/2 2024-08-21T21:54:29.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:54:29.909 INFO:teuthology.orchestra.run.smithi116.stdout:Reinstalled: 2024-08-21T21:54:29.909 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-496.el9.x86_64 2024-08-21T21:54:29.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:54:29.909 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:54:29.998 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-5.14.0-496.el9.x86_64 2/2 2024-08-21T21:54:29.998 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:54:29.999 INFO:teuthology.orchestra.run.smithi178.stdout:Reinstalled: 2024-08-21T21:54:29.999 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-5.14.0-496.el9.x86_64 2024-08-21T21:54:29.999 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:54:29.999 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:54:30.105 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-08-21T21:54:30.164 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-496.el9.x86_64 2024-08-21T21:54:30.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-496.el9.x86_64 2024-08-21T21:54:30.164 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-21T21:54:30.164 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2024-08-21T21:54:30.165 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-21T21:54:30.186 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-08-21T21:54:30.193 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-08-21T21:54:30.221 DEBUG:teuthology.parallel:result is None 2024-08-21T21:54:30.244 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-5.14.0-496.el9.x86_64 2024-08-21T21:54:30.245 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-496.el9.x86_64 2024-08-21T21:54:30.245 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-21T21:54:30.245 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2024-08-21T21:54:30.245 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-21T21:54:30.281 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-08-21T21:54:30.298 DEBUG:teuthology.parallel:result is None 2024-08-21T21:54:30.298 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-21T21:54:30.310 INFO:teuthology.task.internal:Creating test directory... 2024-08-21T21:54:30.311 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-21T21:54:30.313 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-21T21:54:30.353 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-21T21:54:30.359 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-21T21:54:30.365 INFO:teuthology.task.internal:Creating archive directory... 2024-08-21T21:54:30.365 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-21T21:54:30.368 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-21T21:54:30.421 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-21T21:54:30.428 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-21T21:54:30.428 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-21T21:54:30.438 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-21T21:54:30.465 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-21T21:54:30.475 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-21T21:54:30.495 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-21T21:54:30.505 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-21T21:54:30.506 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-21T21:54:30.513 INFO:teuthology.task.internal:Configuring sudo... 2024-08-21T21:54:30.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-21T21:54:30.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-21T21:54:30.571 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-21T21:54:30.579 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-21T21:54:30.579 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-21T21:54:30.583 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-21T21:54:30.626 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-21T21:54:30.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-21T21:54:30.729 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-21T21:54:30.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-21T21:54:30.851 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:54:30.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-21T21:54:30.915 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-21T21:54:30.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-21T21:54:31.008 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-21T21:54:31.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-21T21:54:31.132 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:54:31.132 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-21T21:54:31.197 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-08-21T21:54:31.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-08-21T21:54:31.243 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-21T21:54:31.276 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-21T21:54:31.751 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-21T21:54:31.759 INFO:teuthology.task.internal:Starting timer... 2024-08-21T21:54:31.759 INFO:teuthology.run_tasks:Running task pcp... 2024-08-21T21:54:31.775 INFO:teuthology.run_tasks:Running task selinux... 2024-08-21T21:54:31.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-08-21T21:54:31.845 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-08-21T21:54:31.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-08-21T21:54:31.897 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-08-21T21:54:31.899 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-21T21:54:31.899 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-08-21T21:54:31.917 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-08-21T21:54:31.918 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-08-21T21:54:31.938 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-08-21T21:54:31.938 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-08-21T21:54:31.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-21T21:54:31.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-21T21:54:31.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-21T21:54:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-21T21:54:32.008 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-21T21:54:32.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-08-21T21:54:32.049 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-08-21T21:54:32.090 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-21T21:54:32.100 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-21T21:54:32.110 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-21T21:54:32.111 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-21T21:56:15.218 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-08-21T21:56:15.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-21T21:56:15.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-21T21:56:15.299 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-08-21T21:56:15.382 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-21T21:56:15.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-08-21T21:56:15.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-21T21:56:15.457 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-08-21T21:56:15.539 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-08-21T21:56:15.539 INFO:teuthology.run_tasks:Running task clock... 2024-08-21T21:56:15.550 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-21T21:56:15.550 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-21T21:56:15.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-21T21:56:15.553 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-21T21:56:15.553 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-21T21:56:15.580 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-21T21:56:15.594 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-21T21:56:15.611 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-21T21:56:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-08-21T21:56:15.626 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-21T21:56:15.629 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-08-21T21:56:15.643 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-21T21:56:15.651 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-08-21T21:56:15.657 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-21T21:56:15.663 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-08-21T21:56:15.677 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-21T21:56:15.691 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-21T21:56:15.716 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-08-21T21:56:15.719 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-21T21:56:15.719 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-08-21T21:56:15.719 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.719 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.719 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.719 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.754 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-08-21T21:56:15.757 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-21T21:56:15.757 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-08-21T21:56:15.757 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.757 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.757 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.757 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-21T21:56:15.758 INFO:teuthology.run_tasks:Running task install... 2024-08-21T21:56:15.766 DEBUG:teuthology.task.install:project ceph 2024-08-21T21:56:15.766 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748'}} 2024-08-21T21:56:15.766 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748'} 2024-08-21T21:56:15.767 INFO:teuthology.task.install:Using flavor: default 2024-08-21T21:56:15.772 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-21T21:56:15.772 INFO:teuthology.task.install:extra packages: [] 2024-08-21T21:56:15.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo dnf -y install 'dnf-command(copr)' 2024-08-21T21:56:15.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo dnf -y install 'dnf-command(copr)' 2024-08-21T21:56:16.330 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:43 ago on Wed 21 Aug 2024 09:55:33 PM UTC. 2024-08-21T21:56:16.349 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:43 ago on Wed 21 Aug 2024 09:55:33 PM UTC. 2024-08-21T21:56:16.411 INFO:teuthology.orchestra.run.smithi116.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-08-21T21:56:16.431 INFO:teuthology.orchestra.run.smithi178.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-08-21T21:56:16.453 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:56:16.454 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:56:16.455 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:56:16.473 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:56:16.474 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:56:16.474 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:56:16.523 DEBUG:teuthology.orchestra.run.smithi116:> sudo dnf -y copr enable ceph/el9 2024-08-21T21:56:16.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo dnf -y copr enable ceph/el9 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:of the main distribution, and quality may vary. 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:The Fedora Project does not exercise any power over the contents of 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:, 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:and packages are not held to any quality or security level. 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:Please do not file bug reports about these packages in Fedora 2024-08-21T21:56:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-08-21T21:56:16.709 INFO:teuthology.orchestra.run.smithi178.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-08-21T21:56:16.709 INFO:teuthology.orchestra.run.smithi178.stderr:of the main distribution, and quality may vary. 2024-08-21T21:56:16.709 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-08-21T21:56:16.709 INFO:teuthology.orchestra.run.smithi178.stderr:The Fedora Project does not exercise any power over the contents of 2024-08-21T21:56:16.710 INFO:teuthology.orchestra.run.smithi178.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-08-21T21:56:16.710 INFO:teuthology.orchestra.run.smithi178.stderr:, 2024-08-21T21:56:16.710 INFO:teuthology.orchestra.run.smithi178.stderr:and packages are not held to any quality or security level. 2024-08-21T21:56:16.710 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-08-21T21:56:16.710 INFO:teuthology.orchestra.run.smithi178.stderr:Please do not file bug reports about these packages in Fedora 2024-08-21T21:56:16.710 INFO:teuthology.orchestra.run.smithi178.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-08-21T21:56:16.763 INFO:teuthology.orchestra.run.smithi116.stdout:Repository successfully enabled. 2024-08-21T21:56:16.775 INFO:teuthology.orchestra.run.smithi178.stdout:Repository successfully enabled. 2024-08-21T21:56:16.798 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'tag': None, 'wait_for_package': False} 2024-08-21T21:56:16.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5e746614a5ba6992e5be76bf3df447c931f3748 2024-08-21T21:56:16.810 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'tag': None, 'wait_for_package': False} 2024-08-21T21:56:16.810 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5e746614a5ba6992e5be76bf3df447c931f3748 2024-08-21T21:56:16.949 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/ 2024-08-21T21:56:16.949 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1456.gb5e74661 2024-08-21T21:56:16.952 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/ 2024-08-21T21:56:16.952 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1456.gb5e74661 2024-08-21T21:56:17.061 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-21T21:56:17.062 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:56:17.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-21T21:56:17.067 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-20-1105-quincy/b5e746614a5ba6992e5be76bf3df447c931f3748/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-21T21:56:17.067 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:56:17.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-21T21:56:17.087 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-21T21:56:17.088 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b5e746614a5ba6992e5be76bf3df447c931f3748/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-21T21:56:17.093 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-21T21:56:17.093 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b5e746614a5ba6992e5be76bf3df447c931f3748/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-21T21:56:17.154 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-21T21:56:17.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-21T21:56:17.229 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-21T21:56:17.233 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-21T21:56:17.293 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-08-21T21:56:17.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-08-21T21:56:17.297 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-08-21T21:56:17.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-08-21T21:56:17.520 INFO:teuthology.orchestra.run.smithi178.stdout:45 files removed 2024-08-21T21:56:17.525 INFO:teuthology.orchestra.run.smithi116.stdout:45 files removed 2024-08-21T21:56:17.547 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2024-08-21T21:56:17.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw 2024-08-21T21:56:18.103 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 276 kB/s | 77 kB 00:00 2024-08-21T21:56:18.112 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 272 kB/s | 77 kB 00:00 2024-08-21T21:56:18.387 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 44 kB/s | 11 kB 00:00 2024-08-21T21:56:18.396 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-08-21T21:56:18.671 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 7.2 kB/s | 1.8 kB 00:00 2024-08-21T21:56:18.671 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 7.4 kB/s | 1.8 kB 00:00 2024-08-21T21:56:18.864 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for el9 owned by ceph 48 kB/s | 7.6 kB 00:00 2024-08-21T21:56:18.871 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for el9 owned by ceph 46 kB/s | 7.6 kB 00:00 2024-08-21T21:56:19.305 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 21 MB/s | 8.2 MB 00:00 2024-08-21T21:56:21.190 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-08-21T21:56:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.2 MB 00:03 2024-08-21T21:56:24.465 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-08-21T21:56:25.757 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.6 MB 00:00 2024-08-21T21:56:27.216 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 18 kB 00:00 2024-08-21T21:56:28.129 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 28 MB/s | 22 MB 00:00 2024-08-21T21:56:29.490 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.6 MB 00:01 2024-08-21T21:56:31.116 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 18 kB 00:00 2024-08-21T21:56:31.870 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 35 MB/s | 22 MB 00:00 2024-08-21T21:56:33.460 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-08-21T21:56:35.740 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 14 M 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 3.1 M 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 2.9 M 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 5.0 M 2024-08-21T21:56:35.742 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 22 M 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 25 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 637 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 457 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 4.9 M 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 45 k 2024-08-21T21:56:35.743 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 120 k 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 156 k 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 320 k 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 295 k 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 99 k 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout:Install 22 Packages 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2024-08-21T21:56:35.744 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:56:35.745 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 58 M 2024-08-21T21:56:35.745 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:56:36.652 INFO:teuthology.orchestra.run.smithi178.stdout:(1/24): ceph-base-17.2.7-1456.gb5e74661.el9.x86 8.4 MB/s | 5.0 MB 00:00 2024-08-21T21:56:36.719 INFO:teuthology.orchestra.run.smithi178.stdout:(2/24): ceph-selinux-17.2.7-1456.gb5e74661.el9. 378 kB/s | 25 kB 00:00 2024-08-21T21:56:36.827 INFO:teuthology.orchestra.run.smithi178.stdout:(3/24): libcephfs2-17.2.7-1456.gb5e74661.el9.x8 5.8 MB/s | 637 kB 00:00 2024-08-21T21:56:36.995 INFO:teuthology.orchestra.run.smithi178.stdout:(4/24): ceph-radosgw-17.2.7-1456.gb5e74661.el9. 15 MB/s | 14 MB 00:00 2024-08-21T21:56:37.020 INFO:teuthology.orchestra.run.smithi178.stdout:(5/24): libradosstriper1-17.2.7-1456.gb5e74661. 2.3 MB/s | 457 kB 00:00 2024-08-21T21:56:37.078 INFO:teuthology.orchestra.run.smithi178.stdout:(6/24): python3-ceph-argparse-17.2.7-1456.gb5e7 774 kB/s | 45 kB 00:00 2024-08-21T21:56:37.243 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-21T21:56:37.362 INFO:teuthology.orchestra.run.smithi178.stdout:(7/24): ceph-common-17.2.7-1456.gb5e74661.el9.x 17 MB/s | 22 MB 00:01 2024-08-21T21:56:37.379 INFO:teuthology.orchestra.run.smithi178.stdout:(8/24): python3-ceph-common-17.2.7-1456.gb5e746 400 kB/s | 120 kB 00:00 2024-08-21T21:56:37.454 INFO:teuthology.orchestra.run.smithi178.stdout:(9/24): librgw2-17.2.7-1456.gb5e74661.el9.x86_6 11 MB/s | 4.9 MB 00:00 2024-08-21T21:56:37.471 INFO:teuthology.orchestra.run.smithi178.stdout:(10/24): python3-cephfs-17.2.7-1456.gb5e74661.e 1.4 MB/s | 156 kB 00:00 2024-08-21T21:56:37.496 INFO:teuthology.orchestra.run.smithi178.stdout:(11/24): python3-rados-17.2.7-1456.gb5e74661.el 2.7 MB/s | 320 kB 00:00 2024-08-21T21:56:37.522 INFO:teuthology.orchestra.run.smithi178.stdout:(12/24): python3-rgw-17.2.7-1456.gb5e74661.el9. 1.9 MB/s | 99 kB 00:00 2024-08-21T21:56:37.547 INFO:teuthology.orchestra.run.smithi178.stdout:(13/24): python3-rbd-17.2.7-1456.gb5e74661.el9. 3.1 MB/s | 295 kB 00:00 2024-08-21T21:56:37.856 INFO:teuthology.orchestra.run.smithi178.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 135 kB/s | 45 kB 00:00 2024-08-21T21:56:38.006 INFO:teuthology.orchestra.run.smithi178.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 315 kB/s | 160 kB 00:00 2024-08-21T21:56:38.115 INFO:teuthology.orchestra.run.smithi178.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-08-21T21:56:38.165 INFO:teuthology.orchestra.run.smithi178.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.1 MB/s | 309 kB 00:00 2024-08-21T21:56:38.182 INFO:teuthology.orchestra.run.smithi178.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-08-21T21:56:38.207 INFO:teuthology.orchestra.run.smithi178.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-08-21T21:56:38.241 INFO:teuthology.orchestra.run.smithi178.stdout:(20/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 760 kB/s | 292 kB 00:00 2024-08-21T21:56:38.291 INFO:teuthology.orchestra.run.smithi178.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-08-21T21:56:38.466 INFO:teuthology.orchestra.run.smithi178.stdout:(22/24): librdkafka-1.6.1-102.el9.x86_64.rpm 721 kB/s | 662 kB 00:00 2024-08-21T21:56:38.525 INFO:teuthology.orchestra.run.smithi178.stdout:(23/24): librados2-17.2.7-1456.gb5e74661.el9.x8 11 MB/s | 3.1 MB 00:00 2024-08-21T21:56:38.584 INFO:teuthology.orchestra.run.smithi178.stdout:(24/24): librbd1-17.2.7-1456.gb5e74661.el9.x86_ 10 MB/s | 2.9 MB 00:00 2024-08-21T21:56:38.587 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:56:38.587 INFO:teuthology.orchestra.run.smithi178.stdout:Total 20 MB/s | 58 MB 00:02 2024-08-21T21:56:38.652 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:56:38.687 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:56:38.687 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:56:39.371 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:56:39.372 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:56:39.500 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:56:39.502 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 14 M 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 3.1 M 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 2.9 M 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 5.0 M 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 22 M 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 25 k 2024-08-21T21:56:39.503 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 637 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 457 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 4.9 M 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 45 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 120 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 156 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-21T21:56:39.504 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 320 k 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 295 k 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 99 k 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout:Install 22 Packages 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2024-08-21T21:56:39.505 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:56:39.506 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 58 M 2024-08-21T21:56:39.506 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:56:40.202 INFO:teuthology.orchestra.run.smithi116.stdout:(1/24): ceph-base-17.2.7-1456.gb5e74661.el9.x86 11 MB/s | 5.0 MB 00:00 2024-08-21T21:56:40.261 INFO:teuthology.orchestra.run.smithi116.stdout:(2/24): ceph-selinux-17.2.7-1456.gb5e74661.el9. 433 kB/s | 25 kB 00:00 2024-08-21T21:56:40.344 INFO:teuthology.orchestra.run.smithi116.stdout:(3/24): libcephfs2-17.2.7-1456.gb5e74661.el9.x8 7.5 MB/s | 637 kB 00:00 2024-08-21T21:56:40.428 INFO:teuthology.orchestra.run.smithi116.stdout:(4/24): libradosstriper1-17.2.7-1456.gb5e74661. 5.4 MB/s | 457 kB 00:00 2024-08-21T21:56:40.640 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:56:40.770 INFO:teuthology.orchestra.run.smithi116.stdout:(5/24): librgw2-17.2.7-1456.gb5e74661.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-08-21T21:56:40.912 INFO:teuthology.orchestra.run.smithi116.stdout:(6/24): ceph-radosgw-17.2.7-1456.gb5e74661.el9. 13 MB/s | 14 MB 00:01 2024-08-21T21:56:40.937 INFO:teuthology.orchestra.run.smithi116.stdout:(7/24): python3-ceph-argparse-17.2.7-1456.gb5e7 270 kB/s | 45 kB 00:00 2024-08-21T21:56:40.954 INFO:teuthology.orchestra.run.smithi116.stdout:(8/24): python3-ceph-common-17.2.7-1456.gb5e746 2.8 MB/s | 120 kB 00:00 2024-08-21T21:56:40.979 INFO:teuthology.orchestra.run.smithi116.stdout:(9/24): python3-cephfs-17.2.7-1456.gb5e74661.el 3.6 MB/s | 156 kB 00:00 2024-08-21T21:56:41.004 INFO:teuthology.orchestra.run.smithi116.stdout:(10/24): python3-rados-17.2.7-1456.gb5e74661.el 6.2 MB/s | 320 kB 00:00 2024-08-21T21:56:41.029 INFO:teuthology.orchestra.run.smithi116.stdout:(11/24): python3-rbd-17.2.7-1456.gb5e74661.el9. 5.7 MB/s | 295 kB 00:00 2024-08-21T21:56:41.038 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-21T21:56:41.046 INFO:teuthology.orchestra.run.smithi116.stdout:(12/24): python3-rgw-17.2.7-1456.gb5e74661.el9. 2.3 MB/s | 99 kB 00:00 2024-08-21T21:56:41.086 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 2/26 2024-08-21T21:56:41.178 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 2/26 2024-08-21T21:56:41.255 INFO:teuthology.orchestra.run.smithi116.stdout:(13/24): ceph-common-17.2.7-1456.gb5e74661.el9. 15 MB/s | 22 MB 00:01 2024-08-21T21:56:41.263 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 3/26 2024-08-21T21:56:41.296 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 4/26 2024-08-21T21:56:41.391 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 4/26 2024-08-21T21:56:41.444 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-21T21:56:41.476 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-21T21:56:41.514 INFO:teuthology.orchestra.run.smithi116.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 97 kB/s | 45 kB 00:00 2024-08-21T21:56:41.680 INFO:teuthology.orchestra.run.smithi116.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 246 kB/s | 160 kB 00:00 2024-08-21T21:56:41.781 INFO:teuthology.orchestra.run.smithi116.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 422 kB/s | 42 kB 00:00 2024-08-21T21:56:41.831 INFO:teuthology.orchestra.run.smithi116.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.0 MB/s | 309 kB 00:00 2024-08-21T21:56:41.856 INFO:teuthology.orchestra.run.smithi116.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 854 kB/s | 292 kB 00:00 2024-08-21T21:56:41.882 INFO:teuthology.orchestra.run.smithi116.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 964 kB/s | 49 kB 00:00 2024-08-21T21:56:41.914 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-21T21:56:41.924 INFO:teuthology.orchestra.run.smithi116.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-08-21T21:56:41.941 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 8/26 2024-08-21T21:56:41.957 INFO:teuthology.orchestra.run.smithi116.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 23 MB/s | 1.6 MB 00:00 2024-08-21T21:56:41.987 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 8/26 2024-08-21T21:56:42.016 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-21T21:56:42.033 INFO:teuthology.orchestra.run.smithi116.stdout:(22/24): librdkafka-1.6.1-102.el9.x86_64.rpm 853 kB/s | 662 kB 00:00 2024-08-21T21:56:42.059 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:17.2.7-1456.gb5e74661.el9. 10/26 2024-08-21T21:56:42.104 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 11/26 2024-08-21T21:56:42.158 INFO:teuthology.orchestra.run.smithi116.stdout:(23/24): librados2-17.2.7-1456.gb5e74661.el9.x8 13 MB/s | 3.1 MB 00:00 2024-08-21T21:56:42.166 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 12/26 2024-08-21T21:56:42.194 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_6 13/26 2024-08-21T21:56:42.208 INFO:teuthology.orchestra.run.smithi116.stdout:(24/24): librbd1-17.2.7-1456.gb5e74661.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-08-21T21:56:42.213 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:56:42.213 INFO:teuthology.orchestra.run.smithi116.stdout:Total 21 MB/s | 58 MB 00:02 2024-08-21T21:56:42.260 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_6 13/26 2024-08-21T21:56:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:56:42.300 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-21T21:56:42.310 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:56:42.310 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:56:42.337 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-21T21:56:42.360 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-21T21:56:42.678 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-21T21:56:42.732 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 18/26 2024-08-21T21:56:42.815 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 18/26 2024-08-21T21:56:42.888 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 19/26 2024-08-21T21:56:42.918 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:17.2.7-1456.gb5e74661.el9.x8 20/26 2024-08-21T21:56:42.996 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:56:42.996 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:56:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:56:44.580 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-21T21:56:44.677 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 2/26 2024-08-21T21:56:44.768 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 2/26 2024-08-21T21:56:44.842 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 3/26 2024-08-21T21:56:44.870 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 4/26 2024-08-21T21:56:44.967 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 4/26 2024-08-21T21:56:45.021 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-21T21:56:45.050 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-21T21:56:45.389 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 21/26 2024-08-21T21:56:45.431 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 21/26 2024-08-21T21:56:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-21T21:56:45.514 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 8/26 2024-08-21T21:56:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 8/26 2024-08-21T21:56:45.606 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-21T21:56:45.649 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:17.2.7-1456.gb5e74661.el9. 10/26 2024-08-21T21:56:45.685 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 11/26 2024-08-21T21:56:45.748 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 12/26 2024-08-21T21:56:45.776 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_6 13/26 2024-08-21T21:56:45.842 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_6 13/26 2024-08-21T21:56:45.906 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-21T21:56:45.942 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-21T21:56:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 21/26 2024-08-21T21:56:45.975 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-21T21:56:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 22/26 2024-08-21T21:56:46.098 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 22/26 2024-08-21T21:56:46.098 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-21T21:56:46.098 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-21T21:56:46.098 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:56:46.136 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 23/26 2024-08-21T21:56:46.268 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-21T21:56:46.304 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 18/26 2024-08-21T21:56:46.411 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 18/26 2024-08-21T21:56:46.476 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 19/26 2024-08-21T21:56:46.507 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:17.2.7-1456.gb5e74661.el9.x8 20/26 2024-08-21T21:56:49.078 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 21/26 2024-08-21T21:56:49.112 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 21/26 2024-08-21T21:56:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 21/26 2024-08-21T21:56:49.665 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 22/26 2024-08-21T21:56:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 22/26 2024-08-21T21:56:49.771 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-21T21:56:49.771 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-21T21:56:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:56:49.808 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 23/26 2024-08-21T21:56:56.032 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 23/26 2024-08-21T21:56:56.032 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-08-21T21:56:56.032 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-08-21T21:56:56.032 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-08-21T21:56:56.033 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-08-21T21:56:56.033 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-08-21T21:56:56.033 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2024-08-21T21:56:56.033 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-08-21T21:56:56.033 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:56:57.356 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 24/26 2024-08-21T21:56:57.427 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 24/26 2024-08-21T21:56:57.427 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:56:57.427 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-21T21:56:57.427 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-21T21:56:57.427 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-21T21:56:57.427 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:56:57.428 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-21T21:56:57.501 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-21T21:56:57.501 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-21T21:56:58.784 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-21T21:56:58.784 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 1/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 2/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 3/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 4/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 5/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_6 6/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 7/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1456.gb5e74661.el9. 8/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:17.2.7-1456.gb5e74661.el9.x8 9/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 10/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 11/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 12/26 2024-08-21T21:56:58.785 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 13/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-21T21:56:58.786 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-21T21:56:58.787 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-21T21:56:58.787 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 23/26 2024-08-21T21:56:58.787 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-21T21:56:58.787 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 25/26 2024-08-21T21:56:59.236 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 23/26 2024-08-21T21:56:59.236 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-08-21T21:56:59.236 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-08-21T21:56:59.236 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-08-21T21:56:59.236 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-08-21T21:56:59.236 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-08-21T21:56:59.236 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-08-21T21:56:59.237 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-08-21T21:56:59.237 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:56:59.459 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-21T21:56:59.460 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.461 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:56:59.462 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-21T21:56:59.462 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:56:59.462 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:56:59.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2024-08-21T21:57:00.319 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:27 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:00.455 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 48 M 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 163 k 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-21T21:57:00.456 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-21T21:57:00.457 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:00.457 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:00.457 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:00.457 INFO:teuthology.orchestra.run.smithi178.stdout:Install 4 Packages 2024-08-21T21:57:00.457 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:00.457 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 48 M 2024-08-21T21:57:00.457 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 193 M 2024-08-21T21:57:00.458 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:00.584 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 24/26 2024-08-21T21:57:00.662 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 24/26 2024-08-21T21:57:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-21T21:57:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-21T21:57:00.662 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-21T21:57:00.663 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:00.664 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-21T21:57:00.700 INFO:teuthology.orchestra.run.smithi178.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-08-21T21:57:00.742 INFO:teuthology.orchestra.run.smithi178.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-08-21T21:57:00.754 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-21T21:57:00.754 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-21T21:57:00.792 INFO:teuthology.orchestra.run.smithi178.stdout:(3/4): libcephsqlite-17.2.7-1456.gb5e74661.el9. 691 kB/s | 163 kB 00:00 2024-08-21T21:57:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-21T21:57:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 1/26 2024-08-21T21:57:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 2/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 3/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 4/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 5/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_6 6/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 7/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1456.gb5e74661.el9. 8/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:17.2.7-1456.gb5e74661.el9.x8 9/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 10/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 11/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 12/26 2024-08-21T21:57:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 13/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 23/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-21T21:57:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 25/26 2024-08-21T21:57:02.102 INFO:teuthology.orchestra.run.smithi178.stdout:(4/4): ceph-test-17.2.7-1456.gb5e74661.el9.x86_ 31 MB/s | 48 MB 00:01 2024-08-21T21:57:02.104 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:02.105 INFO:teuthology.orchestra.run.smithi178.stdout:Total 29 MB/s | 48 MB 00:01 2024-08-21T21:57:02.133 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:02.152 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:02.152 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:02.529 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:02.529 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-21T21:57:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:02.703 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-08-21T21:57:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:02.705 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:02.996 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-test 2024-08-21T21:57:03.094 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:03.187 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-21T21:57:03.238 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-21T21:57:03.252 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 3/4 2024-08-21T21:57:03.587 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:26 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:03.722 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 48 M 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 163 k 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-21T21:57:03.723 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:03.724 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:03.724 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:03.724 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-08-21T21:57:03.724 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:03.724 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 48 M 2024-08-21T21:57:03.724 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 193 M 2024-08-21T21:57:03.725 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:03.911 INFO:teuthology.orchestra.run.smithi116.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for http://mirrors.lug.mtu.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 141.219.188.21) 2024-08-21T21:57:04.052 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): libcephsqlite-17.2.7-1456.gb5e74661.el9. 710 kB/s | 163 kB 00:00 2024-08-21T21:57:04.096 INFO:teuthology.orchestra.run.smithi116.stdout:[MIRROR] socat-1.7.4.1-6.el9.x86_64.rpm: Status code: 404 for https://mirrors.lug.mtu.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-6.el9.x86_64.rpm (IP: 141.219.188.21) 2024-08-21T21:57:04.103 INFO:teuthology.orchestra.run.smithi116.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for http://mirrors.lug.mtu.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 141.219.188.21) 2024-08-21T21:57:04.148 INFO:teuthology.orchestra.run.smithi116.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://mirrors.lug.mtu.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 141.219.188.21) 2024-08-21T21:57:04.254 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 706 kB/s | 304 kB 00:00 2024-08-21T21:57:04.288 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 271 kB/s | 64 kB 00:00 2024-08-21T21:57:05.387 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): ceph-test-17.2.7-1456.gb5e74661.el9.x86_ 31 MB/s | 48 MB 00:01 2024-08-21T21:57:05.390 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:05.390 INFO:teuthology.orchestra.run.smithi116.stdout:Total 29 MB/s | 48 MB 00:01 2024-08-21T21:57:05.418 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:05.438 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:05.438 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:05.816 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:05.816 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:06.395 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:06.484 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-21T21:57:06.525 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-21T21:57:06.552 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 3/4 2024-08-21T21:57:07.655 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 3/4 2024-08-21T21:57:07.724 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 4/4 2024-08-21T21:57:10.075 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 4/4 2024-08-21T21:57:10.075 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 1/4 2024-08-21T21:57:10.075 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 2/4 2024-08-21T21:57:10.076 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-21T21:57:10.367 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:10.368 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:10.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2024-08-21T21:57:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 3/4 2024-08-21T21:57:11.026 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 4/4 2024-08-21T21:57:11.194 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:38 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:11.322 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:11.324 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:11.324 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:11.324 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:11.324 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 6.5 k 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 2.1 M 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 1.4 M 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 240 k 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 3.9 M 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 15 M 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-21T21:57:11.325 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-21T21:57:11.326 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-21T21:57:11.327 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-21T21:57:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:11.328 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:11.328 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:11.328 INFO:teuthology.orchestra.run.smithi178.stdout:Install 34 Packages 2024-08-21T21:57:11.328 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:11.329 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 27 M 2024-08-21T21:57:11.329 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 96 M 2024-08-21T21:57:11.329 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:11.987 INFO:teuthology.orchestra.run.smithi178.stdout:(1/34): ceph-17.2.7-1456.gb5e74661.el9.x86_64.r 58 kB/s | 6.5 kB 00:00 2024-08-21T21:57:12.162 INFO:teuthology.orchestra.run.smithi178.stdout:(2/34): ceph-mgr-17.2.7-1456.gb5e74661.el9.x86_ 4.9 MB/s | 1.4 MB 00:00 2024-08-21T21:57:12.212 INFO:teuthology.orchestra.run.smithi178.stdout:(3/34): ceph-mds-17.2.7-1456.gb5e74661.el9.x86_ 6.1 MB/s | 2.1 MB 00:00 2024-08-21T21:57:12.313 INFO:teuthology.orchestra.run.smithi178.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1456.gb5e7 2.3 MB/s | 240 kB 00:00 2024-08-21T21:57:12.371 INFO:teuthology.orchestra.run.smithi178.stdout:(5/34): ceph-mon-17.2.7-1456.gb5e74661.el9.x86_ 10 MB/s | 3.9 MB 00:00 2024-08-21T21:57:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-08-21T21:57:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 3.2 MB/s | 106 kB 00:00 2024-08-21T21:57:12.697 INFO:teuthology.orchestra.run.smithi178.stdout:(8/34): ceph-osd-17.2.7-1456.gb5e74661.el9.x86_ 29 MB/s | 15 MB 00:00 2024-08-21T21:57:12.722 INFO:teuthology.orchestra.run.smithi178.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-08-21T21:57:12.764 INFO:teuthology.orchestra.run.smithi178.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.2 MB 00:00 2024-08-21T21:57:12.789 INFO:teuthology.orchestra.run.smithi178.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-08-21T21:57:12.831 INFO:teuthology.orchestra.run.smithi178.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 932 kB/s | 125 kB 00:00 2024-08-21T21:57:13.057 INFO:teuthology.orchestra.run.smithi178.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-08-21T21:57:13.174 INFO:teuthology.orchestra.run.smithi178.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 225 kB/s | 77 kB 00:00 2024-08-21T21:57:13.232 INFO:teuthology.orchestra.run.smithi178.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 291 kB/s | 51 kB 00:00 2024-08-21T21:57:13.266 INFO:teuthology.orchestra.run.smithi178.stdout:(16/34): python3-mako-1.1.4-6.el9.noarch.rpm 343 kB/s | 172 kB 00:00 2024-08-21T21:57:13.283 INFO:teuthology.orchestra.run.smithi178.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 382 kB/s | 42 kB 00:00 2024-08-21T21:57:13.308 INFO:teuthology.orchestra.run.smithi178.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 575 kB/s | 43 kB 00:00 2024-08-21T21:57:13.333 INFO:teuthology.orchestra.run.smithi178.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 678 kB/s | 11 kB 00:00 2024-08-21T21:57:13.359 INFO:teuthology.orchestra.run.smithi178.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 759 kB/s | 18 kB 00:00 2024-08-21T21:57:13.384 INFO:teuthology.orchestra.run.smithi178.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-08-21T21:57:13.409 INFO:teuthology.orchestra.run.smithi178.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-08-21T21:57:13.426 INFO:teuthology.orchestra.run.smithi178.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 463 kB/s | 19 kB 00:00 2024-08-21T21:57:13.443 INFO:teuthology.orchestra.run.smithi178.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-08-21T21:57:13.460 INFO:teuthology.orchestra.run.smithi178.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-08-21T21:57:13.462 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 4/4 2024-08-21T21:57:13.463 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 1/4 2024-08-21T21:57:13.463 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 2/4 2024-08-21T21:57:13.463 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-21T21:57:13.485 INFO:teuthology.orchestra.run.smithi178.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-08-21T21:57:13.502 INFO:teuthology.orchestra.run.smithi178.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 979 kB/s | 16 kB 00:00 2024-08-21T21:57:13.527 INFO:teuthology.orchestra.run.smithi178.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-08-21T21:57:13.552 INFO:teuthology.orchestra.run.smithi178.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-21T21:57:13.578 INFO:teuthology.orchestra.run.smithi178.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-21T21:57:13.603 INFO:teuthology.orchestra.run.smithi178.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-08-21T21:57:13.628 INFO:teuthology.orchestra.run.smithi178.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 796 kB/s | 20 kB 00:00 2024-08-21T21:57:13.653 INFO:teuthology.orchestra.run.smithi178.stdout:(33/34): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-21T21:57:13.679 INFO:teuthology.orchestra.run.smithi178.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-21T21:57:13.683 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:13.683 INFO:teuthology.orchestra.run.smithi178.stdout:Total 12 MB/s | 27 MB 00:02 2024-08-21T21:57:13.806 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-21T21:57:13.806 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:13.806 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:13.806 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:13.807 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:13.807 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-21T21:57:13.807 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-21T21:57:13.807 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:13.807 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:13.812 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:13.848 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:13.848 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:14.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph 2024-08-21T21:57:14.231 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:14.231 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:14.668 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:37 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:14.804 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 6.5 k 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 2.1 M 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 1.4 M 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 240 k 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 3.9 M 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 15 M 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-21T21:57:14.807 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-21T21:57:14.808 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-21T21:57:14.809 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-21T21:57:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-21T21:57:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-21T21:57:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:14.810 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:14.810 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:14.810 INFO:teuthology.orchestra.run.smithi116.stdout:Install 34 Packages 2024-08-21T21:57:14.810 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:14.811 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 27 M 2024-08-21T21:57:14.811 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 96 M 2024-08-21T21:57:14.811 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:15.475 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:15.529 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-21T21:57:15.559 INFO:teuthology.orchestra.run.smithi116.stdout:(1/34): ceph-17.2.7-1456.gb5e74661.el9.x86_64.r 46 kB/s | 6.5 kB 00:00 2024-08-21T21:57:15.564 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-21T21:57:15.597 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-21T21:57:15.689 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-21T21:57:15.734 INFO:teuthology.orchestra.run.smithi116.stdout:(2/34): ceph-mds-17.2.7-1456.gb5e74661.el9.x86_ 6.5 MB/s | 2.1 MB 00:00 2024-08-21T21:57:15.766 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-21T21:57:15.767 INFO:teuthology.orchestra.run.smithi116.stdout:(3/34): ceph-mgr-17.2.7-1456.gb5e74661.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-08-21T21:57:15.807 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-21T21:57:15.843 INFO:teuthology.orchestra.run.smithi116.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1456.gb5e7 3.1 MB/s | 240 kB 00:00 2024-08-21T21:57:15.843 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-21T21:57:15.872 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-21T21:57:15.898 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-21T21:57:15.923 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-21T21:57:15.955 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-21T21:57:15.960 INFO:teuthology.orchestra.run.smithi116.stdout:(5/34): ceph-mon-17.2.7-1456.gb5e74661.el9.x86_ 9.8 MB/s | 3.9 MB 00:00 2024-08-21T21:57:15.982 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-21T21:57:16.025 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-21T21:57:16.073 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-21T21:57:16.105 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-21T21:57:16.177 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-21T21:57:16.243 INFO:teuthology.orchestra.run.smithi116.stdout:(6/34): ceph-osd-17.2.7-1456.gb5e74661.el9.x86_ 30 MB/s | 15 MB 00:00 2024-08-21T21:57:16.326 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-21T21:57:16.404 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-21T21:57:16.449 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-21T21:57:16.488 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-21T21:57:16.535 INFO:teuthology.orchestra.run.smithi116.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 366 kB/s | 253 kB 00:00 2024-08-21T21:57:16.536 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-21T21:57:16.597 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-21T21:57:16.644 INFO:teuthology.orchestra.run.smithi116.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-08-21T21:57:16.669 INFO:teuthology.orchestra.run.smithi116.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 250 kB/s | 106 kB 00:00 2024-08-21T21:57:16.752 INFO:teuthology.orchestra.run.smithi116.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-08-21T21:57:16.784 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-21T21:57:16.842 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-21T21:57:16.878 INFO:teuthology.orchestra.run.smithi116.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-08-21T21:57:16.911 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-21T21:57:16.936 INFO:teuthology.orchestra.run.smithi116.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 804 kB/s | 215 kB 00:00 2024-08-21T21:57:16.962 INFO:teuthology.orchestra.run.smithi116.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 824 kB/s | 172 kB 00:00 2024-08-21T21:57:16.978 INFO:teuthology.orchestra.run.smithi116.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-08-21T21:57:17.000 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-21T21:57:17.004 INFO:teuthology.orchestra.run.smithi116.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-21T21:57:17.037 INFO:teuthology.orchestra.run.smithi116.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-08-21T21:57:17.047 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-21T21:57:17.054 INFO:teuthology.orchestra.run.smithi116.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-08-21T21:57:17.088 INFO:teuthology.orchestra.run.smithi116.stdout:(18/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-08-21T21:57:17.105 INFO:teuthology.orchestra.run.smithi116.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 726 kB/s | 11 kB 00:00 2024-08-21T21:57:17.130 INFO:teuthology.orchestra.run.smithi116.stdout:(20/34): python3-packaging-20.9-5.el9.noarch.rp 398 kB/s | 77 kB 00:00 2024-08-21T21:57:17.156 INFO:teuthology.orchestra.run.smithi116.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-08-21T21:57:17.164 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-21T21:57:17.189 INFO:teuthology.orchestra.run.smithi116.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 760 kB/s | 19 kB 00:00 2024-08-21T21:57:17.214 INFO:teuthology.orchestra.run.smithi116.stdout:(23/34): python3-jaraco-text-3.2.0-6.el9.noarch 854 kB/s | 20 kB 00:00 2024-08-21T21:57:17.240 INFO:teuthology.orchestra.run.smithi116.stdout:(24/34): python3-jaraco-collections-3.0.0-8.el9 212 kB/s | 23 kB 00:00 2024-08-21T21:57:17.265 INFO:teuthology.orchestra.run.smithi116.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-08-21T21:57:17.299 INFO:teuthology.orchestra.run.smithi116.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-08-21T21:57:17.320 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1456.gb5e74661.el9. 29/34 2024-08-21T21:57:17.324 INFO:teuthology.orchestra.run.smithi116.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-08-21T21:57:17.349 INFO:teuthology.orchestra.run.smithi116.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-08-21T21:57:17.352 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 30/34 2024-08-21T21:57:17.374 INFO:teuthology.orchestra.run.smithi116.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-21T21:57:17.378 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 30/34 2024-08-21T21:57:17.379 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:17.379 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-21T21:57:17.379 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-21T21:57:17.379 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-21T21:57:17.379 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:17.400 INFO:teuthology.orchestra.run.smithi116.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-21T21:57:17.425 INFO:teuthology.orchestra.run.smithi116.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-08-21T21:57:17.450 INFO:teuthology.orchestra.run.smithi116.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-08-21T21:57:17.475 INFO:teuthology.orchestra.run.smithi116.stdout:(33/34): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-21T21:57:17.501 INFO:teuthology.orchestra.run.smithi116.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-08-21T21:57:17.505 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:17.505 INFO:teuthology.orchestra.run.smithi116.stdout:Total 10 MB/s | 27 MB 00:02 2024-08-21T21:57:17.637 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:17.674 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:17.674 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:18.055 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:18.056 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:18.687 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 31/34 2024-08-21T21:57:18.714 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 31/34 2024-08-21T21:57:18.714 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:18.714 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-21T21:57:18.715 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-21T21:57:18.715 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-21T21:57:18.715 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:19.087 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 32/34 2024-08-21T21:57:19.111 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 32/34 2024-08-21T21:57:19.112 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:19.112 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-21T21:57:19.112 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-21T21:57:19.112 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-21T21:57:19.112 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:19.289 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:19.321 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 33/34 2024-08-21T21:57:19.345 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 33/34 2024-08-21T21:57:19.346 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:19.346 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-21T21:57:19.346 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-21T21:57:19.346 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-21T21:57:19.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:19.370 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-21T21:57:19.402 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-21T21:57:19.408 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 34/34 2024-08-21T21:57:19.435 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-21T21:57:19.527 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-21T21:57:19.603 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-21T21:57:19.645 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-21T21:57:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-21T21:57:19.702 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-21T21:57:19.728 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-21T21:57:19.761 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-21T21:57:19.802 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-21T21:57:19.836 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-21T21:57:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-21T21:57:19.920 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-21T21:57:19.951 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-21T21:57:20.024 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-21T21:57:20.161 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-21T21:57:20.250 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-21T21:57:20.304 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-21T21:57:20.343 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-21T21:57:20.391 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-21T21:57:20.451 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-21T21:57:20.630 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-21T21:57:20.680 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-21T21:57:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-21T21:57:20.846 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-21T21:57:20.892 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-21T21:57:21.009 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 34/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 1/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 2/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 3/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 4/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 5/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1456.gb5e74661.el9. 6/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-21T21:57:21.110 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-21T21:57:21.111 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-21T21:57:21.112 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-21T21:57:21.112 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-21T21:57:21.113 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-21T21:57:21.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1456.gb5e74661.el9. 29/34 2024-08-21T21:57:21.190 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 30/34 2024-08-21T21:57:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 30/34 2024-08-21T21:57:21.215 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:21.215 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-21T21:57:21.215 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-21T21:57:21.215 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-21T21:57:21.215 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:21.986 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-21T21:57:21.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-21T21:57:21.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-21T21:57:21.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-21T21:57:21.989 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:21.989 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:22.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2024-08-21T21:57:22.574 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 31/34 2024-08-21T21:57:22.605 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 31/34 2024-08-21T21:57:22.606 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:22.606 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-21T21:57:22.606 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-21T21:57:22.606 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-21T21:57:22.606 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:22.883 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:49 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:22.968 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:57:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 32/34 2024-08-21T21:57:22.999 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 32/34 2024-08-21T21:57:22.999 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:22.999 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-21T21:57:22.999 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-21T21:57:22.999 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-21T21:57:23.000 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:23.008 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:23.009 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:57:23.010 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:23.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2024-08-21T21:57:23.209 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 33/34 2024-08-21T21:57:23.233 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 33/34 2024-08-21T21:57:23.233 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:23.233 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-21T21:57:23.233 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-21T21:57:23.233 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-21T21:57:23.234 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:23.295 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 34/34 2024-08-21T21:57:23.637 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:50 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:23.763 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 83 k 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:57:23.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:23.765 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 83 k 2024-08-21T21:57:23.765 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 375 k 2024-08-21T21:57:23.765 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:23.954 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-17.2.7-1456.gb5e74661.el9.noarch.rpm 437 kB/s | 83 kB 00:00 2024-08-21T21:57:23.955 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:23.955 INFO:teuthology.orchestra.run.smithi178.stdout:Total 433 kB/s | 83 kB 00:00 2024-08-21T21:57:23.955 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:23.960 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:23.960 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:23.962 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:23.962 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:24.035 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:24.495 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:24.580 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:24.826 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 34/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 1/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 2/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 3/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 4/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 5/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1456.gb5e74661.el9. 6/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-21T21:57:24.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-21T21:57:24.828 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-21T21:57:24.829 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-21T21:57:24.829 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-21T21:57:24.829 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-21T21:57:24.829 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-21T21:57:24.829 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-21T21:57:24.830 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-21T21:57:24.830 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-21T21:57:24.830 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-21T21:57:24.830 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-21T21:57:24.830 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-21T21:57:24.906 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:25.140 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:25.141 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:25.141 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:25.141 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:25.141 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:25.141 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:25.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2024-08-21T21:57:25.884 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-21T21:57:25.884 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:25.884 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:25.884 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:25.885 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:25.885 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:25.885 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:25.885 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:25.885 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:25.885 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-21T21:57:25.886 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-21T21:57:25.886 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-21T21:57:25.886 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-21T21:57:25.886 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-21T21:57:25.886 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-21T21:57:25.886 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-21T21:57:25.887 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-21T21:57:25.887 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-21T21:57:25.887 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-21T21:57:25.887 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-21T21:57:25.887 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-21T21:57:25.887 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-21T21:57:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-21T21:57:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-21T21:57:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-21T21:57:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-08-21T21:57:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-21T21:57:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-21T21:57:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-21T21:57:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-21T21:57:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-21T21:57:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-21T21:57:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-21T21:57:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-21T21:57:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-21T21:57:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-21T21:57:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-21T21:57:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:25.890 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:25.951 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:52 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:26.080 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:26.081 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-08-21T21:57:26.081 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:26.081 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:26.081 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 137 k 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 137 k 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 379 k 2024-08-21T21:57:26.082 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:26.257 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-base 2024-08-21T21:57:26.285 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-17.2.7-1456.gb5e746 677 kB/s | 137 kB 00:00 2024-08-21T21:57:26.286 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:26.286 INFO:teuthology.orchestra.run.smithi178.stdout:Total 671 kB/s | 137 kB 00:00 2024-08-21T21:57:26.286 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:26.294 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:26.294 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:26.351 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:26.352 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:26.490 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:26.536 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661. 1/1 2024-08-21T21:57:26.562 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661. 1/1 2024-08-21T21:57:26.562 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:26.562 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-21T21:57:26.562 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:26.823 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:49 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:26.907 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-base-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:57:26.949 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:26.950 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:57:26.950 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:27.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install cephadm 2024-08-21T21:57:27.490 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661. 1/1 2024-08-21T21:57:27.491 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:27.491 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:27.491 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:27.491 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:27.491 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:27.574 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:50 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:27.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2024-08-21T21:57:27.699 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:27.700 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:27.700 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:57:27.700 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:27.700 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:27.700 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 83 k 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 83 k 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 375 k 2024-08-21T21:57:27.701 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:27.900 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm-17.2.7-1456.gb5e74661.el9.noarch.rpm 417 kB/s | 83 kB 00:00 2024-08-21T21:57:27.900 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:27.900 INFO:teuthology.orchestra.run.smithi116.stdout:Total 414 kB/s | 83 kB 00:00 2024-08-21T21:57:27.900 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:27.905 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:27.906 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:27.907 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:27.908 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:27.987 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:28.266 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:55 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:28.355 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:57:28.398 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:28.398 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:57:28.398 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:28.432 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:28.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2024-08-21T21:57:28.501 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:28.842 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:29.029 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:56 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:29.102 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:57:29.102 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:29.102 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:29.103 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:29.103 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:29.103 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:29.157 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 3.4 M 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 24 k 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 15 k 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-21T21:57:29.158 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout:Install 7 Packages 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:29.159 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.7 M 2024-08-21T21:57:29.160 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 80 M 2024-08-21T21:57:29.160 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:29.304 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-immutable-object-cache 2024-08-21T21:57:29.480 INFO:teuthology.orchestra.run.smithi178.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1456.gb5e7 94 kB/s | 15 kB 00:00 2024-08-21T21:57:29.506 INFO:teuthology.orchestra.run.smithi178.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1456.gb5e 135 kB/s | 24 kB 00:00 2024-08-21T21:57:29.547 INFO:teuthology.orchestra.run.smithi178.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-08-21T21:57:29.572 INFO:teuthology.orchestra.run.smithi178.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-08-21T21:57:29.598 INFO:teuthology.orchestra.run.smithi178.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-08-21T21:57:29.623 INFO:teuthology.orchestra.run.smithi178.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-08-21T21:57:29.698 INFO:teuthology.orchestra.run.smithi178.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1456.gb5e74661 9.0 MB/s | 3.4 MB 00:00 2024-08-21T21:57:29.702 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:29.702 INFO:teuthology.orchestra.run.smithi178.stdout:Total 6.7 MB/s | 3.7 MB 00:00 2024-08-21T21:57:29.702 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:29.710 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:29.710 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:29.789 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:29.790 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:29.882 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:52 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:30.011 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 137 k 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:30.012 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:57:30.013 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:30.013 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 137 k 2024-08-21T21:57:30.013 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 379 k 2024-08-21T21:57:30.013 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:30.088 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:30.158 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-08-21T21:57:30.192 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-08-21T21:57:30.222 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-immutable-object-cache-17.2.7-1456.gb5e746 659 kB/s | 137 kB 00:00 2024-08-21T21:57:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:Total 654 kB/s | 137 kB 00:00 2024-08-21T21:57:30.223 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:30.231 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:30.231 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:30.238 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-08-21T21:57:30.270 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-08-21T21:57:30.286 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:30.286 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:30.299 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1456.gb5e74661.el9.n 5/7 2024-08-21T21:57:30.431 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:30.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661. 1/1 2024-08-21T21:57:30.523 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661. 1/1 2024-08-21T21:57:30.523 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:57:30.523 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-21T21:57:30.523 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:31.290 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1456.gb5e74661.el9. 6/7 2024-08-21T21:57:31.328 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarc 7/7 2024-08-21T21:57:31.420 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661. 1/1 2024-08-21T21:57:31.421 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:31.421 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:31.421 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:57:31.421 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:31.421 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:31.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr 2024-08-21T21:57:32.238 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:55 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:32.324 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-mgr-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:57:32.359 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarc 7/7 2024-08-21T21:57:32.359 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1456.gb5e74661.el9. 1/7 2024-08-21T21:57:32.359 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarc 2/7 2024-08-21T21:57:32.359 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1456.gb5e74661.el9.n 3/7 2024-08-21T21:57:32.359 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-08-21T21:57:32.359 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-08-21T21:57:32.359 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-08-21T21:57:32.366 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:32.367 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:57:32.367 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:32.433 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-dashboard 2024-08-21T21:57:32.648 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-08-21T21:57:32.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:32.648 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:32.648 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:32.648 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:32.648 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:32.649 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-08-21T21:57:32.649 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-08-21T21:57:32.649 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-21T21:57:32.649 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-21T21:57:32.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:32.649 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:32.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-21T21:57:32.994 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:55 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:33.120 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:33.121 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:33.121 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:57:33.121 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:33.121 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 3.4 M 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 24 k 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 15 k 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:33.122 INFO:teuthology.orchestra.run.smithi116.stdout:Install 7 Packages 2024-08-21T21:57:33.123 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:33.123 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.7 M 2024-08-21T21:57:33.123 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 80 M 2024-08-21T21:57:33.123 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:33.409 INFO:teuthology.orchestra.run.smithi116.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1456.gb5e 178 kB/s | 24 kB 00:00 2024-08-21T21:57:33.413 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:00 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:33.426 INFO:teuthology.orchestra.run.smithi116.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1456.gb5e7 95 kB/s | 15 kB 00:00 2024-08-21T21:57:33.468 INFO:teuthology.orchestra.run.smithi116.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2024-08-21T21:57:33.493 INFO:teuthology.orchestra.run.smithi116.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-08-21T21:57:33.519 INFO:teuthology.orchestra.run.smithi116.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-08-21T21:57:33.543 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:33.571 INFO:teuthology.orchestra.run.smithi116.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-08-21T21:57:33.572 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================= 2024-08-21T21:57:33.572 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:33.572 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================= 2024-08-21T21:57:33.572 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:33.572 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 7.4 M 2024-08-21T21:57:33.572 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:57:33.572 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.26-1.el9 appstream 38 k 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.26-1.el9 appstream 4.9 M 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.19-7.el9 appstream 245 k 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:33.573 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:33.574 INFO:teuthology.orchestra.run.smithi178.stdout:======================================================================================= 2024-08-21T21:57:33.574 INFO:teuthology.orchestra.run.smithi178.stdout:Install 12 Packages 2024-08-21T21:57:33.574 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:33.574 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 38 M 2024-08-21T21:57:33.574 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 212 M 2024-08-21T21:57:33.574 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:33.652 INFO:teuthology.orchestra.run.smithi116.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1456.gb5e74661 8.8 MB/s | 3.4 MB 00:00 2024-08-21T21:57:33.656 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:33.656 INFO:teuthology.orchestra.run.smithi116.stdout:Total 6.8 MB/s | 3.7 MB 00:00 2024-08-21T21:57:33.656 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:33.664 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:33.664 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:33.756 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:33.757 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:33.798 INFO:teuthology.orchestra.run.smithi178.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-08-21T21:57:33.840 INFO:teuthology.orchestra.run.smithi178.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 801 kB 00:00 2024-08-21T21:57:33.923 INFO:teuthology.orchestra.run.smithi178.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-08-21T21:57:33.974 INFO:teuthology.orchestra.run.smithi178.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-08-21T21:57:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:34.082 INFO:teuthology.orchestra.run.smithi178.stdout:(5/12): openblas-0.3.26-1.el9.x86_64.rpm 354 kB/s | 38 kB 00:00 2024-08-21T21:57:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-08-21T21:57:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-08-21T21:57:34.182 INFO:teuthology.orchestra.run.smithi178.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-14 14 MB/s | 7.4 MB 00:00 2024-08-21T21:57:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-08-21T21:57:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-08-21T21:57:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1456.gb5e74661.el9.n 5/7 2024-08-21T21:57:34.383 INFO:teuthology.orchestra.run.smithi178.stdout:(7/12): python3-devel-3.9.19-7.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-08-21T21:57:34.775 INFO:teuthology.orchestra.run.smithi178.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2024-08-21T21:57:34.858 INFO:teuthology.orchestra.run.smithi178.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-08-21T21:57:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1456.gb5e74661.el9. 6/7 2024-08-21T21:57:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarc 7/7 2024-08-21T21:57:35.342 INFO:teuthology.orchestra.run.smithi178.stdout:(10/12): openblas-openmp-0.3.26-1.el9.x86_64.rp 3.9 MB/s | 4.9 MB 00:01 2024-08-21T21:57:35.600 INFO:teuthology.orchestra.run.smithi178.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-08-21T21:57:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarc 7/7 2024-08-21T21:57:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1456.gb5e74661.el9. 1/7 2024-08-21T21:57:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarc 2/7 2024-08-21T21:57:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1456.gb5e74661.el9.n 3/7 2024-08-21T21:57:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-08-21T21:57:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-08-21T21:57:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-08-21T21:57:36.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-08-21T21:57:36.713 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:36.713 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:36.714 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:36.954 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-21T21:57:37.317 INFO:teuthology.orchestra.run.smithi178.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-08-21T21:57:37.318 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:37.319 INFO:teuthology.orchestra.run.smithi178.stdout:Total 10 MB/s | 38 MB 00:03 2024-08-21T21:57:37.509 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:00 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:37.637 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:37.637 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-08-21T21:57:37.637 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 7.4 M 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.26-1.el9 appstream 38 k 2024-08-21T21:57:37.638 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.26-1.el9 appstream 4.9 M 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.19-7.el9 appstream 245 k 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-08-21T21:57:37.639 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:37.640 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 38 M 2024-08-21T21:57:37.640 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 212 M 2024-08-21T21:57:37.640 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:37.681 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:37.707 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:37.707 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:38.044 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-08-21T21:57:38.160 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-08-21T21:57:38.180 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:38.181 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:38.302 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-14 16 MB/s | 7.4 MB 00:00 2024-08-21T21:57:38.320 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 119 kB/s | 33 kB 00:00 2024-08-21T21:57:38.495 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): openblas-0.3.26-1.el9.x86_64.rpm 219 kB/s | 38 kB 00:00 2024-08-21T21:57:38.553 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 73 kB/s | 18 kB 00:00 2024-08-21T21:57:38.955 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:38.978 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): python3-devel-3.9.19-7.el9.x86_64.rpm 576 kB/s | 245 kB 00:00 2024-08-21T21:57:39.029 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-21T21:57:39.074 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-08-21T21:57:39.174 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-08-21T21:57:39.226 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.19-7.el9.x86_64 4/12 2024-08-21T21:57:39.411 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.26-1.el9.x86_64 5/12 2024-08-21T21:57:39.442 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.26-1.el9.x86_64 6/12 2024-08-21T21:57:39.528 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-21T21:57:39.721 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.9 MB/s | 3.0 MB 00:01 2024-08-21T21:57:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-08-21T21:57:40.138 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): openblas-openmp-0.3.26-1.el9.x86_64.rp 3.0 MB/s | 4.9 MB 00:01 2024-08-21T21:57:40.181 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-21T21:57:40.296 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-21T21:57:41.543 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-21T21:57:41.796 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.8 MB/s | 5.1 MB 00:02 2024-08-21T21:57:42.580 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-21T21:57:42.600 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74 12/12 2024-08-21T21:57:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.7 MB/s | 16 MB 00:03 2024-08-21T21:57:43.322 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:43.323 INFO:teuthology.orchestra.run.smithi116.stdout:Total 6.7 MB/s | 38 MB 00:05 2024-08-21T21:57:43.681 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:43.707 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:43.707 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:44.193 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:44.193 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:44.942 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:45.011 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-21T21:57:45.061 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-08-21T21:57:45.163 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-08-21T21:57:45.195 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.19-7.el9.x86_64 4/12 2024-08-21T21:57:45.304 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74 12/12 2024-08-21T21:57:45.304 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74 1/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.26-1.el9.x86_64 7/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.26-1.el9.x86_64 8/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.19-7.el9.x86_64 9/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-21T21:57:45.305 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-21T21:57:45.378 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.26-1.el9.x86_64 5/12 2024-08-21T21:57:45.411 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.26-1.el9.x86_64 6/12 2024-08-21T21:57:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.26-1.el9.x86_64 2024-08-21T21:57:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.26-1.el9.x86_64 2024-08-21T21:57:45.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.19-7.el9.x86_64 2024-08-21T21:57:45.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-21T21:57:45.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-21T21:57:45.725 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-21T21:57:45.725 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:45.725 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:46.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2024-08-21T21:57:46.141 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-21T21:57:46.266 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-21T21:57:46.693 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:13 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:46.832 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:46.833 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:46.833 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:46.833 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:46.833 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:46.833 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 47 k 2024-08-21T21:57:46.833 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:57:46.833 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-21T21:57:46.834 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:46.835 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:46.835 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:46.835 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-08-21T21:57:46.835 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:46.835 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 2.2 M 2024-08-21T21:57:46.835 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 26 M 2024-08-21T21:57:46.835 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:47.159 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): ceph-mgr-rook-17.2.7-1456.gb5e74661.el9 293 kB/s | 47 kB 00:00 2024-08-21T21:57:47.251 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 75 kB/s | 19 kB 00:00 2024-08-21T21:57:47.268 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 95 kB/s | 26 kB 00:00 2024-08-21T21:57:47.510 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 614 kB/s | 159 kB 00:00 2024-08-21T21:57:47.527 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-21T21:57:47.610 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 815 kB/s | 279 kB 00:00 2024-08-21T21:57:47.635 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 427 kB/s | 54 kB 00:00 2024-08-21T21:57:47.678 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 429 kB/s | 222 kB 00:00 2024-08-21T21:57:47.703 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 348 kB/s | 32 kB 00:00 2024-08-21T21:57:47.737 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-08-21T21:57:47.762 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-08-21T21:57:47.787 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-08-21T21:57:47.812 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-google-auth-2.33.0-1.el9.noarc 1.6 MB/s | 220 kB 00:00 2024-08-21T21:57:47.854 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-08-21T21:57:47.856 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:47.856 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-08-21T21:57:47.920 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:47.934 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:47.934 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:48.020 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:48.021 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:48.537 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:48.565 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-21T21:57:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74 12/12 2024-08-21T21:57:48.702 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-21T21:57:48.764 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-21T21:57:48.819 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-21T21:57:48.848 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-21T21:57:48.890 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-21T21:57:48.991 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-21T21:57:49.093 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-21T21:57:49.141 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-21T21:57:49.852 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-21T21:57:49.897 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-21T21:57:49.930 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-21T21:57:49.975 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-21T21:57:50.000 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 13/13 2024-08-21T21:57:50.711 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 13/13 2024-08-21T21:57:50.711 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 1/13 2024-08-21T21:57:50.711 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-21T21:57:50.712 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-21T21:57:51.045 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-21T21:57:51.045 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:51.045 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:51.045 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:51.045 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-21T21:57:51.046 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:51.047 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:51.095 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74 12/12 2024-08-21T21:57:51.095 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74 1/12 2024-08-21T21:57:51.095 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-08-21T21:57:51.095 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.26-1.el9.x86_64 7/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.26-1.el9.x86_64 8/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.19-7.el9.x86_64 9/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-21T21:57:51.096 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-21T21:57:51.262 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2024-08-21T21:57:51.512 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.26-1.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.26-1.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.19-7.el9.x86_64 2024-08-21T21:57:51.513 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-21T21:57:51.514 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-21T21:57:51.514 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-21T21:57:51.514 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:51.514 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:51.822 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:18 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:51.912 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-rook 2024-08-21T21:57:51.951 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:51.951 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:51.951 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:51.951 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:51.951 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 124 k 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-08-21T21:57:51.952 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:51.953 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 7.0 M 2024-08-21T21:57:51.953 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 33 M 2024-08-21T21:57:51.953 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:52.264 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-08-21T21:57:52.322 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-08-21T21:57:52.348 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-08-21T21:57:52.373 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-08-21T21:57:52.415 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): ceph-mgr-cephadm-17.2.7-1456.gb5e74661.e 322 kB/s | 124 kB 00:00 2024-08-21T21:57:52.484 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:15 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:52.617 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:52.618 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:52.618 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 47 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-21T21:57:52.619 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2024-08-21T21:57:52.620 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:52.621 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.2 M 2024-08-21T21:57:52.621 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2024-08-21T21:57:52.621 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:52.887 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 279 kB/s | 26 kB 00:00 2024-08-21T21:57:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 164 kB/s | 19 kB 00:00 2024-08-21T21:57:52.971 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): ceph-mgr-rook-17.2.7-1456.gb5e74661.el9 270 kB/s | 47 kB 00:00 2024-08-21T21:57:53.046 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-21T21:57:53.096 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 222 kB 00:00 2024-08-21T21:57:53.122 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-08-21T21:57:53.163 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-08-21T21:57:53.189 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-08-21T21:57:53.222 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-08-21T21:57:53.248 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-08-21T21:57:53.273 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-08-21T21:57:53.298 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): python3-google-auth-2.33.0-1.el9.noarc 1.6 MB/s | 220 kB 00:00 2024-08-21T21:57:53.340 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-08-21T21:57:53.342 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:53.343 INFO:teuthology.orchestra.run.smithi116.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-08-21T21:57:53.407 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:53.423 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:53.423 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:53.440 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-08-21T21:57:53.442 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:53.442 INFO:teuthology.orchestra.run.smithi178.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-08-21T21:57:53.511 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:53.512 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:53.513 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:53.523 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:53.524 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:53.620 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:53.621 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:53.926 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:54.028 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:54.088 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-21T21:57:54.142 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-21T21:57:54.193 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-21T21:57:54.249 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-21T21:57:54.304 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-21T21:57:54.334 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-21T21:57:54.375 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-21T21:57:54.476 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-21T21:57:54.578 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-21T21:57:54.635 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-21T21:57:54.659 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-21T21:57:54.744 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-21T21:57:54.800 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-21T21:57:54.821 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 6/6 2024-08-21T21:57:55.351 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-21T21:57:55.407 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-21T21:57:55.441 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-21T21:57:55.478 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-21T21:57:55.494 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 13/13 2024-08-21T21:57:55.575 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 6/6 2024-08-21T21:57:55.575 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/6 2024-08-21T21:57:55.575 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-21T21:57:55.575 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-21T21:57:55.575 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-21T21:57:55.575 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-21T21:57:55.826 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:55.827 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:56.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2024-08-21T21:57:56.203 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 13/13 2024-08-21T21:57:56.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 1/13 2024-08-21T21:57:56.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-21T21:57:56.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-21T21:57:56.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-21T21:57:56.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-21T21:57:56.527 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:56.528 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:57:56.627 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:23 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:56.755 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 813 k 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-21T21:57:56.756 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:56.757 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:56.757 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:56.757 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-08-21T21:57:56.757 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:56.757 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 893 k 2024-08-21T21:57:56.757 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 2.6 M 2024-08-21T21:57:56.758 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:56.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-cephadm 2024-08-21T21:57:57.158 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 541 kB/s | 80 kB 00:00 2024-08-21T21:57:57.275 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): ceph-fuse-17.2.7-1456.gb5e74661.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-08-21T21:57:57.276 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:57.276 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.7 MB/s | 893 kB 00:00 2024-08-21T21:57:57.288 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:57.308 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:57.308 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:57.340 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:20 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:57:57.386 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:57.387 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:57:57.468 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 124 k 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-21T21:57:57.469 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-21T21:57:57.470 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:57.470 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:57:57.470 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:57:57.470 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-08-21T21:57:57.470 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:57:57.470 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 7.0 M 2024-08-21T21:57:57.470 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 33 M 2024-08-21T21:57:57.471 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:57:57.575 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:57:57.720 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-21T21:57:57.781 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 2/2 2024-08-21T21:57:57.917 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-08-21T21:57:57.975 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-08-21T21:57:58.000 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): ceph-mgr-cephadm-17.2.7-1456.gb5e74661.e 569 kB/s | 124 kB 00:00 2024-08-21T21:57:58.026 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-08-21T21:57:58.068 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-08-21T21:57:58.376 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-08-21T21:57:58.379 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:58.379 INFO:teuthology.orchestra.run.smithi116.stdout:Total 7.7 MB/s | 7.0 MB 00:00 2024-08-21T21:57:58.391 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 2/2 2024-08-21T21:57:58.391 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/2 2024-08-21T21:57:58.449 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:57:58.460 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:57:58.460 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:57:58.557 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:57:58.558 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:57:58.645 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-21T21:57:58.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:58.645 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:57:58.645 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-21T21:57:58.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:58.645 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:57:58.860 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:57:58.920 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-volume 2024-08-21T21:57:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-21T21:57:59.077 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-21T21:57:59.493 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:26 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:57:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-21T21:57:59.626 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 269 k 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:57:59.627 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:57:59.628 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:57:59.628 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 269 k 2024-08-21T21:57:59.628 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 1.3 M 2024-08-21T21:57:59.628 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:57:59.680 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-21T21:57:59.736 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-21T21:57:59.765 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 6/6 2024-08-21T21:57:59.838 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-volume-17.2.7-1456.gb5e74661.el9.noarch.rp 1.2 MB/s | 269 kB 00:00 2024-08-21T21:57:59.839 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:57:59.839 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.2 MB/s | 269 kB 00:00 2024-08-21T21:57:59.839 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:57:59.847 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:57:59.847 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:57:59.871 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:57:59.872 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:58:00.086 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:58:00.149 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:58:00.165 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:58:00.165 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:58:00.165 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-21T21:58:00.165 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:00.480 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 6/6 2024-08-21T21:58:00.481 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 1/6 2024-08-21T21:58:00.481 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-21T21:58:00.481 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-21T21:58:00.481 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-21T21:58:00.481 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-21T21:58:00.758 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-21T21:58:00.758 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:00.759 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:00.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-fuse 2024-08-21T21:58:01.117 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:58:01.117 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:01.117 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:58:01.117 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:58:01.117 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:01.117 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:01.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2024-08-21T21:58:01.565 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:24 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:01.692 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:01.692 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:58:01.692 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:01.692 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:58:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 813 k 2024-08-21T21:58:01.692 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 893 k 2024-08-21T21:58:01.693 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 2.6 M 2024-08-21T21:58:01.694 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:58:01.965 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:28 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:02.053 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): ceph-fuse-17.2.7-1456.gb5e74661.el9.x86_ 3.1 MB/s | 813 kB 00:00 2024-08-21T21:58:02.097 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:02.097 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 126 k 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 126 k 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 448 k 2024-08-21T21:58:02.098 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:58:02.277 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-17.2.7-1456.gb5e74661.el9.x86_64 706 kB/s | 126 kB 00:00 2024-08-21T21:58:02.278 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:02.278 INFO:teuthology.orchestra.run.smithi178.stdout:Total 701 kB/s | 126 kB 00:00 2024-08-21T21:58:02.278 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:58:02.283 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:58:02.284 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:58:02.347 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:58:02.347 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:58:02.495 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:58:02.520 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 110 kB/s | 80 kB 00:00 2024-08-21T21:58:02.521 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:02.521 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.1 MB/s | 893 kB 00:00 2024-08-21T21:58:02.533 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:58:02.554 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:58:02.554 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:58:02.590 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:02.634 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:58:02.634 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:58:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:58:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-21T21:58:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 2/2 2024-08-21T21:58:03.031 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:03.267 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:03.267 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:03.267 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:58:03.267 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:03.267 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:03.267 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:03.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2024-08-21T21:58:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 2/2 2024-08-21T21:58:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/2 2024-08-21T21:58:03.900 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-21T21:58:03.900 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:03.900 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:03.900 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-21T21:58:03.900 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:03.900 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:04.102 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:31 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:04.162 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-volume 2024-08-21T21:58:04.189 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:04.232 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:04.233 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:58:04.234 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:04.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2024-08-21T21:58:04.737 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:27 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:04.866 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:04.867 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:04.867 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:17.2.7-1456.gb5e74661.el9 ceph-noarch 269 k 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 269 k 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 1.3 M 2024-08-21T21:58:04.868 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:58:04.909 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:31 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:05.040 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:05.041 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:05.041 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-08-21T21:58:05.041 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:05.041 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:58:05.041 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 25 k 2024-08-21T21:58:05.041 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:05.041 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:58:05.042 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:05.042 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:58:05.042 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:05.042 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 25 k 2024-08-21T21:58:05.042 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 110 k 2024-08-21T21:58:05.042 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:58:05.083 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-volume-17.2.7-1456.gb5e74661.el9.noarch.rp 1.2 MB/s | 269 kB 00:00 2024-08-21T21:58:05.083 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:05.084 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.2 MB/s | 269 kB 00:00 2024-08-21T21:58:05.084 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:58:05.091 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:58:05.091 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:58:05.117 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:58:05.117 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:58:05.189 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-17.2.7-1456.gb5e74661.el9.x86_6 167 kB/s | 25 kB 00:00 2024-08-21T21:58:05.190 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:05.190 INFO:teuthology.orchestra.run.smithi178.stdout:Total 165 kB/s | 25 kB 00:00 2024-08-21T21:58:05.190 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:58:05.193 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:58:05.193 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:58:05.195 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:58:05.195 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:58:05.280 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:58:05.319 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:58:05.385 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:58:05.392 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:05.401 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:58:05.401 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:58:05.401 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-21T21:58:05.401 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:05.739 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:05.950 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:05.950 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:05.950 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:58:05.950 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:05.950 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:05.950 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:06.113 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2024-08-21T21:58:06.358 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 1/1 2024-08-21T21:58:06.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:06.359 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:06.359 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:17.2.7-1456.gb5e74661.el9.noarch 2024-08-21T21:58:06.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:06.359 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:06.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados-devel 2024-08-21T21:58:06.681 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:33 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:06.767 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:06.809 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:06.810 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:58:06.810 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:06.871 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2024-08-21T21:58:07.217 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:30 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:07.344 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 126 k 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:07.345 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:58:07.346 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:07.346 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 126 k 2024-08-21T21:58:07.346 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 448 k 2024-08-21T21:58:07.346 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:58:07.437 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:34 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:07.523 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:07.538 INFO:teuthology.orchestra.run.smithi116.stdout:librados-devel-17.2.7-1456.gb5e74661.el9.x86_64 658 kB/s | 126 kB 00:00 2024-08-21T21:58:07.539 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:07.539 INFO:teuthology.orchestra.run.smithi116.stdout:Total 653 kB/s | 126 kB 00:00 2024-08-21T21:58:07.539 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:58:07.544 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:58:07.544 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:58:07.565 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:07.566 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:58:07.567 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:07.606 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:58:07.607 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:58:07.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2024-08-21T21:58:07.760 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:58:07.850 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:08.197 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:35 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:08.283 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:08.285 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:08.326 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:58:08.327 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:08.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2024-08-21T21:58:08.516 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:08.516 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:08.516 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:08.516 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:08.516 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:08.516 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:08.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs2 2024-08-21T21:58:08.957 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:35 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:09.041 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:09.084 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:09.085 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:58:09.085 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:09.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2024-08-21T21:58:09.316 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:32 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:09.403 INFO:teuthology.orchestra.run.smithi116.stdout:Package libcephfs2-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:09.445 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:09.446 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:58:09.446 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:09.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs-devel 2024-08-21T21:58:09.709 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:36 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:09.796 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:09.839 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:09.840 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:58:09.841 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:09.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2024-08-21T21:58:10.078 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:33 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:10.206 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 25 k 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:10.207 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 25 k 2024-08-21T21:58:10.208 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 110 k 2024-08-21T21:58:10.208 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:58:10.366 INFO:teuthology.orchestra.run.smithi116.stdout:libcephfs-devel-17.2.7-1456.gb5e74661.el9.x86_6 157 kB/s | 25 kB 00:00 2024-08-21T21:58:10.367 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:10.367 INFO:teuthology.orchestra.run.smithi116.stdout:Total 156 kB/s | 25 kB 00:00 2024-08-21T21:58:10.367 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:58:10.370 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:58:10.370 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:58:10.372 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:58:10.372 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:58:10.459 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:58:10.467 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:37 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:10.553 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:10.568 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:10.595 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:10.596 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-08-21T21:58:10.596 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:10.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2024-08-21T21:58:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:11.122 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:11.122 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:11.123 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:11.123 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:11.123 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:11.123 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:11.228 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:38 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:11.309 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados2 2024-08-21T21:58:11.354 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:11.355 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:11.355 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-08-21T21:58:11.355 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:11.355 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 82 k 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 82 k 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 222 k 2024-08-21T21:58:11.356 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:58:11.541 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-17.2.7-1456.gb5e74661.el9.x86_64.rpm 441 kB/s | 82 kB 00:00 2024-08-21T21:58:11.542 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:11.542 INFO:teuthology.orchestra.run.smithi178.stdout:Total 438 kB/s | 82 kB 00:00 2024-08-21T21:58:11.542 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:58:11.549 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:58:11.549 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:58:11.614 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:58:11.614 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:58:11.752 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:58:11.845 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:11.870 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:34 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:11.954 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:11.996 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:11.997 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:58:11.997 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:12.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librbd1 2024-08-21T21:58:12.198 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:12.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:12.473 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:58:12.473 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:12.473 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:12.630 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:35 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:12.715 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:12.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2024-08-21T21:58:12.757 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:12.758 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:58:12.758 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:12.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rados 2024-08-21T21:58:13.321 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:40 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:13.392 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:36 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:13.455 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:13.456 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:13.456 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-08-21T21:58:13.456 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:13.456 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 2.9 M 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 2.9 M 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 12 M 2024-08-21T21:58:13.457 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:58:13.477 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rados-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:13.519 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:13.520 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:58:13.520 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:13.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rgw 2024-08-21T21:58:13.803 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-17.2.7-1456.gb5e74661.el9.x86_64.rpm 8.5 MB/s | 2.9 MB 00:00 2024-08-21T21:58:13.804 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:13.804 INFO:teuthology.orchestra.run.smithi178.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-08-21T21:58:13.805 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:58:13.814 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:58:13.814 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:58:13.892 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:58:13.893 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:58:14.153 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:37 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:14.238 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rgw-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:14.280 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:14.281 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:58:14.281 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:14.305 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:58:14.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-cephfs 2024-08-21T21:58:14.371 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:14.395 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:14.396 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:58:14.396 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-21T21:58:14.396 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-21T21:58:14.396 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-21T21:58:14.396 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:14.909 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:37 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:14.995 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-cephfs-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:15.037 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:15.038 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:58:15.039 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:15.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rbd 2024-08-21T21:58:15.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:15.474 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:15.474 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:58:15.474 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:15.474 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:15.474 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:15.682 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:38 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:15.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2024-08-21T21:58:15.766 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rbd-2:17.2.7-1456.gb5e74661.el9.x86_64 is already installed. 2024-08-21T21:58:15.810 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:15.811 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-21T21:58:15.811 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:15.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-fuse 2024-08-21T21:58:16.285 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:43 ago on Wed 21 Aug 2024 09:56:33 PM UTC. 2024-08-21T21:58:16.417 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-08-21T21:58:16.418 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:16.418 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-08-21T21:58:16.418 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:16.418 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-08-21T21:58:16.418 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 165 k 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 165 k 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 477 k 2024-08-21T21:58:16.419 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-08-21T21:58:16.456 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:39 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:16.584 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 82 k 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:58:16.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:16.586 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 82 k 2024-08-21T21:58:16.586 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 222 k 2024-08-21T21:58:16.586 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:58:16.640 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-17.2.7-1456.gb5e74661.el9.x86_64.rpm 746 kB/s | 165 kB 00:00 2024-08-21T21:58:16.640 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:16.641 INFO:teuthology.orchestra.run.smithi178.stdout:Total 742 kB/s | 165 kB 00:00 2024-08-21T21:58:16.641 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-08-21T21:58:16.649 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-08-21T21:58:16.649 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-08-21T21:58:16.712 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-08-21T21:58:16.713 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-08-21T21:58:16.769 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-fuse-17.2.7-1456.gb5e74661.el9.x86_64.rpm 446 kB/s | 82 kB 00:00 2024-08-21T21:58:16.770 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:16.771 INFO:teuthology.orchestra.run.smithi116.stdout:Total 443 kB/s | 82 kB 00:00 2024-08-21T21:58:16.771 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:58:16.777 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:58:16.777 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:58:16.839 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:58:16.839 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:58:16.865 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-08-21T21:58:16.964 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:16.988 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:58:17.089 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:17.448 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:17.474 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:17.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:17.714 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:17.714 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:17.714 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:17.714 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:17.714 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:17.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:17.741 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:17.741 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-08-21T21:58:17.741 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:17.741 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:17.741 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-08-21T21:58:17.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2024-08-21T21:58:17.988 DEBUG:teuthology.parallel:result is None 2024-08-21T21:58:18.525 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:41 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:18.658 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 2.9 M 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.9 M 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 12 M 2024-08-21T21:58:18.659 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:58:18.997 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-mirror-17.2.7-1456.gb5e74661.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-08-21T21:58:18.998 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:18.998 INFO:teuthology.orchestra.run.smithi116.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-08-21T21:58:18.998 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:58:19.007 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:58:19.007 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:58:19.085 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:58:19.086 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:58:19.495 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:58:19.556 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:19.581 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:19.581 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-21T21:58:19.581 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-21T21:58:19.581 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-21T21:58:19.581 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-21T21:58:19.582 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:20.664 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:20.664 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:20.664 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:20.664 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:20.664 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:20.665 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:20.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-nbd 2024-08-21T21:58:21.525 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:44 ago on Wed 21 Aug 2024 09:56:37 PM UTC. 2024-08-21T21:58:21.653 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-21T21:58:21.653 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:21.653 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-08-21T21:58:21.653 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:21.654 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-21T21:58:21.654 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:17.2.7-1456.gb5e74661.el9 ceph 165 k 2024-08-21T21:58:21.654 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:21.654 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-21T21:58:21.654 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-21T21:58:21.654 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-21T21:58:21.654 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:21.655 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 165 k 2024-08-21T21:58:21.655 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 477 k 2024-08-21T21:58:21.655 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-21T21:58:21.859 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-nbd-17.2.7-1456.gb5e74661.el9.x86_64.rpm 810 kB/s | 165 kB 00:00 2024-08-21T21:58:21.859 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-21T21:58:21.860 INFO:teuthology.orchestra.run.smithi116.stdout:Total 804 kB/s | 165 kB 00:00 2024-08-21T21:58:21.860 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-21T21:58:21.867 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-21T21:58:21.867 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-21T21:58:21.930 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-21T21:58:21.930 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-21T21:58:22.079 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-21T21:58:22.170 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:22.546 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:22.790 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 1/1 2024-08-21T21:58:22.791 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:22.791 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-21T21:58:22.791 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:17.2.7-1456.gb5e74661.el9.x86_64 2024-08-21T21:58:22.791 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:22.791 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-21T21:58:23.100 DEBUG:teuthology.parallel:result is None 2024-08-21T21:58:23.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5e746614a5ba6992e5be76bf3df447c931f3748 2024-08-21T21:58:23.245 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-21T21:58:23.266 INFO:teuthology.orchestra.run.smithi116.stdout:17.2.7-1456.gb5e74661.el9 2024-08-21T21:58:23.267 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1456.gb5e74661.el9 2024-08-21T21:58:23.267 INFO:teuthology.task.install:The correct ceph version 17.2.7-1456.gb5e74661 is installed. 2024-08-21T21:58:23.269 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b5e746614a5ba6992e5be76bf3df447c931f3748 2024-08-21T21:58:23.405 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-21T21:58:23.432 INFO:teuthology.orchestra.run.smithi178.stdout:17.2.7-1456.gb5e74661.el9 2024-08-21T21:58:23.432 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1456.gb5e74661.el9 2024-08-21T21:58:23.432 INFO:teuthology.task.install:The correct ceph version 17.2.7-1456.gb5e74661 is installed. 2024-08-21T21:58:23.434 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-21T21:58:23.434 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:23.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-21T21:58:23.460 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:23.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-21T21:58:23.503 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-21T21:58:23.503 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:23.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-08-21T21:58:23.528 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-21T21:58:23.593 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:23.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-08-21T21:58:23.622 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-21T21:58:23.691 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-21T21:58:23.691 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:23.692 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-21T21:58:23.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-21T21:58:23.780 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:23.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-21T21:58:23.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-21T21:58:23.876 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-21T21:58:23.876 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:23.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-08-21T21:58:23.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-21T21:58:23.967 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:23.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-08-21T21:58:23.994 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-21T21:58:24.061 INFO:teuthology.run_tasks:Running task ceph... 2024-08-21T21:58:24.136 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-21T21:58:24.136 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-08-21T21:58:24.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /var/log/ceph 2024-08-21T21:58:24.165 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-21T21:58:24.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-21T21:58:24.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-21T21:58:24.231 INFO:tasks.ceph:Creating extra log directories... 2024-08-21T21:58:24.231 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-21T21:58:24.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-21T21:58:24.305 INFO:tasks.ceph:Creating ceph cluster c1... 2024-08-21T21:58:24.305 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-21T21:58:24.306 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274', 'branch': 'wip-yuri7-testing-2024-08-20-1105-quincy', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7866274', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2024-08-20-1105-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri7-testing-2024-08-20-1105-quincy', 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6565, 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri7-testing-2024-08-20-1105-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMZw3gcuLo5X2pqLvNADs7gKZN4so6LJethSQlhnXNg1d9qR1MWdWYmyGuH5Ktr0PrbjgDDhWg5WK2aczrvW/zA=', 'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBwiuhM1fPYG8eHoRh/WKgq0Bvla0NGDeBYIL52Cb/n8CRt+ckwfBetifQpkEOPmoN0phJKjpnMu2pcB0AwETrs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '4d4c5e30374204f3b32f867e61bb4e1a3880f672', 'timestamp': '2024-08-21_20:32:34', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-21T21:58:24.306 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-08-21T21:58:24.334 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2024-08-21T21:58:24.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-08-21T21:58:24.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2024-08-21T21:58:24.418 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:24.418 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-08-21T21:58:24.468 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-21T21:58:24.468 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-08-21T21:58:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-21T21:58:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:24.523 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-21T21:58:24.524 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:24.524 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:24.524 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-21 21:58:19.828795128 +0000 2024-08-21T21:58:24.524 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-21 21:55:54.389523386 +0000 2024-08-21T21:58:24.524 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-21 21:55:54.389523386 +0000 2024-08-21T21:58:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-21 21:55:54.389523386 +0000 2024-08-21T21:58:24.524 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-21T21:58:24.587 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-21T21:58:24.587 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-21T21:58:24.587 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000179581 s, 2.9 MB/s 2024-08-21T21:58:24.588 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-21T21:58:24.644 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-21 21:58:19.828795128 +0000 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-21 21:55:54.714517971 +0000 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-21 21:55:54.714517971 +0000 2024-08-21T21:58:24.700 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-21 21:55:54.714517971 +0000 2024-08-21T21:58:24.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-21T21:58:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-21T21:58:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-21T21:58:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000204822 s, 2.5 MB/s 2024-08-21T21:58:24.764 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-21T21:58:24.820 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-08-21T21:58:24.875 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-21T21:58:24.875 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:24.875 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-08-21T21:58:24.875 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:24.875 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:24.875 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-21 21:58:19.829795112 +0000 2024-08-21T21:58:24.876 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-21 21:55:55.041512522 +0000 2024-08-21T21:58:24.876 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-21 21:55:55.041512522 +0000 2024-08-21T21:58:24.876 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-21 21:55:55.041512522 +0000 2024-08-21T21:58:24.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-21T21:58:24.938 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-21T21:58:24.938 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-21T21:58:24.938 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000185493 s, 2.8 MB/s 2024-08-21T21:58:24.939 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-21T21:58:24.995 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-08-21T21:58:25.050 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-21T21:58:25.050 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:25.050 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-08-21T21:58:25.050 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:25.050 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:25.050 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-21 21:58:19.829795112 +0000 2024-08-21T21:58:25.051 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-21 21:55:55.359507223 +0000 2024-08-21T21:58:25.051 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-21 21:55:55.359507223 +0000 2024-08-21T21:58:25.051 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-21 21:55:55.359507223 +0000 2024-08-21T21:58:25.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-21T21:58:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-21T21:58:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-21T21:58:25.113 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000192197 s, 2.7 MB/s 2024-08-21T21:58:25.115 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-21T21:58:25.171 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:58:25.171 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-21T21:58:25.171 INFO:tasks.ceph:Generating config... 2024-08-21T21:58:25.172 INFO:tasks.ceph:[client] debug rgw = 20 2024-08-21T21:58:25.172 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] setgroup = ceph 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client] setuser = ceph 2024-08-21T21:58:25.173 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-08-21T21:58:25.173 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-21T21:58:25.173 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-21T21:58:25.173 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-21T21:58:25.173 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-21T21:58:25.174 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-21T21:58:25.174 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-21T21:58:25.174 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-21T21:58:25.174 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-08-21T21:58:25.174 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-08-21T21:58:25.174 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-08-21T21:58:25.174 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-08-21T21:58:25.174 INFO:tasks.ceph:Setting up c1.mon.a... 2024-08-21T21:58:25.175 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2024-08-21T21:58:25.258 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/c1.keyring 2024-08-21T21:58:25.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2024-08-21T21:58:25.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-08-21T21:58:25.362 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.116')] 2024-08-21T21:58:25.363 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-08-21T21:58:25.364 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:25.364 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-21T21:58:25.417 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --print /home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:25.493 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:58:25.493 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:58:25.493 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid e8a4fd3f-20d2-4a73-ba88-4d584db938fc 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = octopus 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:fsid e8a4fd3f-20d2-4a73-ba88-4d584db938fc 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2024-08-21T21:58:25.493124+0000 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-08-21T21:58:25.493124+0000 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 15 (octopus) 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-08-21T21:58:25.494 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2024-08-21T21:58:25.496 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-21T21:58:25.550 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID e8a4fd3f-20d2-4a73-ba88-4d584db938fc... 2024-08-21T21:58:25.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-08-21T21:58:25.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.633 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-08-21T21:58:25.634 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-21T21:58:25.635 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = 172.21.15.116 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = e8a4fd3f-20d2-4a73-ba88-4d584db938fc 2024-08-21T21:58:25.636 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.637 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.638 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: rocksdb delete range threshold = 0 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.639 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-08-21T21:58:25.640 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-21T21:58:25.641 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: debug rgw = 20 2024-08-21T21:58:25.642 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt require ssl = False 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt s3 kms backend = testing 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw curl low speed time = 300 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw data log num shards = 4 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw md log max shards = 4 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw sync log trim interval = 0 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw sync meta inject err probability = 0.1 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw sync obj etag verify = True 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: setgroup = ceph 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: setuser = ceph 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: rgw data notify interval msec = 0 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-08-21T21:58:25.643 INFO:teuthology.orchestra.run.smithi116.stdout: osd_max_omap_entries_per_request = 10 2024-08-21T21:58:25.644 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-08-21T21:58:25.644 INFO:teuthology.orchestra.run.smithi116.stdout: osd_max_omap_entries_per_request = 1000 2024-08-21T21:58:25.644 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-08-21T21:58:25.644 INFO:teuthology.orchestra.run.smithi116.stdout: osd_max_omap_entries_per_request = 10000 2024-08-21T21:58:25.644 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-08-21T21:58:25.644 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-08-21T21:58:25.644 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-21T21:58:25.645 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-08-21T21:58:25.646 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = 172.21.15.116 2024-08-21T21:58:25.647 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e8a4fd3f-20d2-4a73-ba88-4d584db938fc 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-08-21T21:58:25.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-08-21T21:58:25.649 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: rocksdb delete range threshold = 0 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-08-21T21:58:25.650 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-08-21T21:58:25.651 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-08-21T21:58:25.652 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: debug rgw = 20 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw crypt require ssl = False 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw crypt s3 kms backend = testing 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw curl low speed time = 300 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw data log num shards = 4 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw md log max shards = 4 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw sync log trim interval = 0 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw sync meta inject err probability = 0.1 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: rgw sync obj etag verify = True 2024-08-21T21:58:25.653 INFO:teuthology.orchestra.run.smithi178.stdout: setgroup = ceph 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout: setuser = ceph 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout: rgw data notify interval msec = 0 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout: osd_max_omap_entries_per_request = 10 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout: osd_max_omap_entries_per_request = 1000 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-08-21T21:58:25.654 INFO:teuthology.orchestra.run.smithi178.stdout: osd_max_omap_entries_per_request = 10000 2024-08-21T21:58:25.656 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2024-08-21T21:58:25.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2024-08-21T21:58:25.745 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-21T21:58:25.745 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:25.745 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2024-08-21T21:58:25.760 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:25.760 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2024-08-21T21:58:25.815 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T21:58:25.815 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:25.815 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/c1.keyring 2024-08-21T21:58:25.815 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-08-21T21:58:25.889 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:25.890 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:25.946 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T21:58:25.946 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:25.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/c1.keyring 2024-08-21T21:58:25.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/c1.keyring 2024-08-21T21:58:25.985 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:25.985 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:26.042 INFO:tasks.ceph:Setting up mon nodes... 2024-08-21T21:58:26.042 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-21T21:58:26.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2024-08-21T21:58:26.090 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2024-08-21T21:58:26.092 INFO:tasks.ceph:Setting up mds nodes... 2024-08-21T21:58:26.092 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-21T21:58:26.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2024-08-21T21:58:26.129 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/c1.client.0.keyring 2024-08-21T21:58:26.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2024-08-21T21:58:26.218 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/c1.client.1.keyring 2024-08-21T21:58:26.230 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-21T21:58:26.230 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-21T21:58:26.231 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2024-08-21T21:58:26.295 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:58:26.295 INFO:tasks.ceph:role: c1.osd.0 2024-08-21T21:58:26.295 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T21:58:26.295 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-21T21:58:26.367 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-21T21:58:26.866 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-08-21T21:58:26.868 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-08-21T21:58:26.869 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2024-08-21T21:58:27.014 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2024-08-21T21:58:27.049 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2024-08-21T21:58:27.114 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:58:27.115 INFO:tasks.ceph:role: c1.osd.1 2024-08-21T21:58:27.115 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T21:58:27.115 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-21T21:58:27.186 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-21T21:58:27.186 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-21T21:58:27.187 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-21T21:58:27.643 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-08-21T21:58:27.645 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-08-21T21:58:27.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2024-08-21T21:58:27.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2024-08-21T21:58:27.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2024-08-21T21:58:27.861 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:58:27.861 INFO:tasks.ceph:role: c1.osd.2 2024-08-21T21:58:27.861 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T21:58:27.861 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-21T21:58:27.933 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-21T21:58:27.933 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-21T21:58:27.933 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-21T21:58:27.934 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-21T21:58:27.934 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-21T21:58:27.934 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-08-21T21:58:27.934 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-21T21:58:27.934 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-21T21:58:27.934 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-21T21:58:27.934 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-21T21:58:28.420 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-08-21T21:58:28.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-08-21T21:58:28.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2024-08-21T21:58:28.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2024-08-21T21:58:28.587 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:28.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:28.677+0000 7f1f16632440 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2024-08-21T21:58:28.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:28.683+0000 7f1f16632440 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2024-08-21T21:58:28.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:28.683+0000 7f1f16632440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-08-21T21:58:28.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:28.683+0000 7f1f16632440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-08-21T21:58:28.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:28.683+0000 7f1f16632440 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2024-08-21T21:58:28.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:28.690+0000 7f1f16632440 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2024-08-21T21:58:29.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:29.214+0000 7f1f16632440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:58:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:29.975+0000 7f1f16632440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:58:30.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2024-08-21T21:58:30.793 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:30.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:30.879+0000 7ff95972f440 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2024-08-21T21:58:30.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:30.879+0000 7ff95972f440 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2024-08-21T21:58:30.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:30.879+0000 7ff95972f440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-08-21T21:58:30.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:30.879+0000 7ff95972f440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-08-21T21:58:30.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:30.879+0000 7ff95972f440 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2024-08-21T21:58:30.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:30.879+0000 7ff95972f440 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2024-08-21T21:58:31.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:31.404+0000 7ff95972f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:58:32.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:32.178+0000 7ff95972f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:58:32.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2024-08-21T21:58:32.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:33.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:33.085+0000 7fb42171d440 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2024-08-21T21:58:33.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:33.085+0000 7fb42171d440 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2024-08-21T21:58:33.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:33.085+0000 7fb42171d440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-08-21T21:58:33.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:33.085+0000 7fb42171d440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-08-21T21:58:33.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:33.085+0000 7fb42171d440 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2024-08-21T21:58:33.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:33.086+0000 7fb42171d440 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2024-08-21T21:58:33.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:33.611+0000 7fb42171d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:58:34.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:58:34.374+0000 7fb42171d440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:58:35.160 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2024-08-21T21:58:35.190 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-21T21:58:35.191 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:35.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2024-08-21T21:58:35.258 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:35.258 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2024-08-21T21:58:35.325 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:35.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2024-08-21T21:58:35.391 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:35.391 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2024-08-21T21:58:35.456 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:35.456 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2024-08-21T21:58:35.515 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:35.515 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2024-08-21T21:58:35.571 INFO:tasks.ceph:Adding keys to all mons... 2024-08-21T21:58:35.571 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/c1.keyring 2024-08-21T21:58:35.633 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-08-21T21:58:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQACY8ZmTt1VBRAAZOuROnZ3qEj4gCxAjBhs2w== 2024-08-21T21:58:35.633 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAEY8ZmXuBuKBAAYXPBRMzut5M1FoUaJbC69w== 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAGY8ZmMFVyNBAAqhsJGKUMQW+FsUZ5HiHLvg== 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAJY8ZmltAjBRAAghF27BP47vQ0AE3JF0rTzQ== 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQACY8ZmqfKwBxAAeImMaMES5XfDvf3QAe/70A== 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2024-08-21T21:58:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQACY8ZmQYv+DBAAzHu2nISnJO43YAPlQSZnAQ== 2024-08-21T21:58:35.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-21T21:58:35.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-21T21:58:35.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-21T21:58:35.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-21T21:58:35.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-21T21:58:35.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-21T21:58:35.957 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-21T21:58:35.957 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2024-08-21T21:58:35.981 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2024-08-21T21:58:36.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2024-08-21T21:58:36.096 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/c1.monmap 2024-08-21T21:58:36.151 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2024-08-21T21:58:36.152 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2024-08-21T21:58:36.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2024-08-21T21:58:36.193 INFO:tasks.ceph.c1.mon.a:Started 2024-08-21T21:58:36.193 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2024-08-21T21:58:36.194 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2024-08-21T21:58:36.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2024-08-21T21:58:36.196 INFO:tasks.ceph.c1.mgr.x:Started 2024-08-21T21:58:36.196 INFO:tasks.ceph:Setting crush tunables to default 2024-08-21T21:58:36.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster c1 osd crush tunables default 2024-08-21T21:58:36.357 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-08-21T21:58:36.368 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2024-08-21T21:58:36.368 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:36.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2024-08-21T21:58:36.392 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:36.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2024-08-21T21:58:36.457 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:36.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2024-08-21T21:58:36.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster c1 osd new 7ade91d4-cc6f-46b1-b954-63937c222eda 0 2024-08-21T21:58:36.731 INFO:teuthology.orchestra.run.smithi116.stdout:0 2024-08-21T21:58:36.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster c1 osd new ade3b803-599d-48f1-9ef3-4ace0e8a41f4 1 2024-08-21T21:58:36.912 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-08-21T21:58:36.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster c1 osd new 03cf29d4-35d2-4971-a6a5-9cdfef6d93b1 2 2024-08-21T21:58:37.007 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.006+0000 7f75a8fbeec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-21T21:58:37.095 INFO:teuthology.orchestra.run.smithi116.stdout:2 2024-08-21T21:58:37.106 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2024-08-21T21:58:37.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2024-08-21T21:58:37.108 INFO:tasks.ceph.c1.osd.0:Started 2024-08-21T21:58:37.108 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2024-08-21T21:58:37.109 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2024-08-21T21:58:37.111 INFO:tasks.ceph.c1.osd.1:Started 2024-08-21T21:58:37.111 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2024-08-21T21:58:37.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2024-08-21T21:58:37.113 INFO:tasks.ceph.c1.osd.2:Started 2024-08-21T21:58:37.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-08-21T21:58:37.125 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.124+0000 7f75a8fbeec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-21T21:58:37.287 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.286+0000 7f75a8fbeec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-21T21:58:37.304 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:37.304 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"fsid":"e8a4fd3f-20d2-4a73-ba88-4d584db938fc","created":"2024-08-21T21:58:36.273783+0000","modified":"2024-08-21T21:58:37.092938+0000","last_up_change":"0.000000","last_in_change":"2024-08-21T21:58:37.092938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7ade91d4-cc6f-46b1-b954-63937c222eda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ade3b803-599d-48f1-9ef3-4ace0e8a41f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03cf29d4-35d2-4971-a6a5-9cdfef6d93b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:58:37.315 INFO:tasks.ceph.ceph_manager.c1:[] 2024-08-21T21:58:37.315 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-21T21:58:37.378 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.376+0000 7f75a8fbeec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-21T21:58:37.524 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.522+0000 7f75a8fbeec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-21T21:58:37.618 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-08-21T21:58:37.628 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.626+0000 7f75a8fbeec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-21T21:58:37.676 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.675+0000 7f75a8fbeec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-21T21:58:37.723 INFO:tasks.ceph.c1.osd.2.smithi116.stderr:2024-08-21T21:58:37.722+0000 7f66ff428440 -1 Falling back to public interface 2024-08-21T21:58:37.727 INFO:tasks.ceph.c1.osd.0.smithi116.stderr:2024-08-21T21:58:37.725+0000 7f53f5213440 -1 Falling back to public interface 2024-08-21T21:58:37.735 INFO:tasks.ceph.c1.osd.1.smithi116.stderr:2024-08-21T21:58:37.733+0000 7f3c16a08440 -1 Falling back to public interface 2024-08-21T21:58:37.739 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.738+0000 7f75a8fbeec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-21T21:58:37.771 INFO:teuthology.misc.health.smithi116.stdout: 2024-08-21T21:58:37.771 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":5,"fsid":"e8a4fd3f-20d2-4a73-ba88-4d584db938fc","created":"2024-08-21T21:58:36.273783+0000","modified":"2024-08-21T21:58:37.092938+0000","last_up_change":"0.000000","last_in_change":"2024-08-21T21:58:37.092938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7ade91d4-cc6f-46b1-b954-63937c222eda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ade3b803-599d-48f1-9ef3-4ace0e8a41f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03cf29d4-35d2-4971-a6a5-9cdfef6d93b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:58:37.781 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-21T21:58:37.833 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.831+0000 7f75a8fbeec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-21T21:58:37.872 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.870+0000 7f75a8fbeec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-21T21:58:37.964 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:37.962+0000 7f75a8fbeec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-21T21:58:38.008 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:38.006+0000 7f75a8fbeec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-21T21:58:38.048 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:38.046+0000 7f75a8fbeec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-21T21:58:39.131 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.130+0000 7f75a8fbeec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-21T21:58:39.179 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.177+0000 7f75a8fbeec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-21T21:58:39.227 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.226+0000 7f75a8fbeec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-21T21:58:39.277 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.275+0000 7f75a8fbeec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-21T21:58:39.371 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.369+0000 7f75a8fbeec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-21T21:58:39.413 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.412+0000 7f75a8fbeec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-21T21:58:39.498 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.496+0000 7f75a8fbeec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-21T21:58:39.886 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.884+0000 7f75a8fbeec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-21T21:58:39.933 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:39.931+0000 7f75a8fbeec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-21T21:58:40.061 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:40.060+0000 7f75a8fbeec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-21T21:58:40.110 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:40.109+0000 7f75a8fbeec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-21T21:58:40.424 INFO:tasks.ceph.c1.osd.1.smithi116.stderr:2024-08-21T21:58:40.423+0000 7f3c16a08440 -1 osd.1 0 log_to_monitors true 2024-08-21T21:58:40.447 INFO:tasks.ceph.c1.osd.2.smithi116.stderr:2024-08-21T21:58:40.445+0000 7f66ff428440 -1 osd.2 0 log_to_monitors true 2024-08-21T21:58:40.462 INFO:tasks.ceph.c1.osd.0.smithi116.stderr:2024-08-21T21:58:40.461+0000 7f53f5213440 -1 osd.0 0 log_to_monitors true 2024-08-21T21:58:41.118 INFO:tasks.ceph.c1.mgr.x.smithi116.stderr:2024-08-21T21:58:41.116+0000 7f758d60c640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-21T21:58:43.377 INFO:tasks.ceph.c1.osd.0.smithi116.stderr:2024-08-21T21:58:43.375+0000 7f53ef9f6640 -1 osd.0 0 waiting for initial osdmap 2024-08-21T21:58:43.383 INFO:tasks.ceph.c1.osd.0.smithi116.stderr:2024-08-21T21:58:43.382+0000 7f53e89e8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-21T21:58:43.419 INFO:tasks.ceph.c1.osd.2.smithi116.stderr:2024-08-21T21:58:43.418+0000 7f66f9bf6640 -1 osd.2 0 waiting for initial osdmap 2024-08-21T21:58:43.423 INFO:tasks.ceph.c1.osd.2.smithi116.stderr:2024-08-21T21:58:43.422+0000 7f66f2be8640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-21T21:58:43.433 INFO:tasks.ceph.c1.osd.1.smithi116.stderr:2024-08-21T21:58:43.432+0000 7f3c111f6640 -1 osd.1 0 waiting for initial osdmap 2024-08-21T21:58:43.437 INFO:tasks.ceph.c1.osd.1.smithi116.stderr:2024-08-21T21:58:43.435+0000 7f3c0a1e8640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-21T21:58:44.085 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-08-21T21:58:44.343 INFO:teuthology.misc.health.smithi116.stdout: 2024-08-21T21:58:44.344 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":7,"fsid":"e8a4fd3f-20d2-4a73-ba88-4d584db938fc","created":"2024-08-21T21:58:36.273783+0000","modified":"2024-08-21T21:58:42.172437+0000","last_up_change":"0.000000","last_in_change":"2024-08-21T21:58:37.092938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"7ade91d4-cc6f-46b1-b954-63937c222eda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ade3b803-599d-48f1-9ef3-4ace0e8a41f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"03cf29d4-35d2-4971-a6a5-9cdfef6d93b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:58:44.355 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-21T21:58:50.658 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2024-08-21T21:58:50.920 INFO:teuthology.misc.health.smithi116.stdout: 2024-08-21T21:58:50.920 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":12,"fsid":"e8a4fd3f-20d2-4a73-ba88-4d584db938fc","created":"2024-08-21T21:58:36.273783+0000","modified":"2024-08-21T21:58:48.384651+0000","last_up_change":"2024-08-21T21:58:44.377019+0000","last_in_change":"2024-08-21T21:58:37.092938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-21T21:58:46.142988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7ade91d4-cc6f-46b1-b954-63937c222eda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":19885},{"type":"v1","addr":"172.21.15.116:6809","nonce":19885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":19885},{"type":"v1","addr":"172.21.15.116:6811","nonce":19885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":19885},{"type":"v1","addr":"172.21.15.116:6815","nonce":19885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":19885},{"type":"v1","addr":"172.21.15.116:6813","nonce":19885}]},"public_addr":"172.21.15.116:6809/19885","cluster_addr":"172.21.15.116:6811/19885","heartbeat_back_addr":"172.21.15.116:6815/19885","heartbeat_front_addr":"172.21.15.116:6813/19885","state":["exists","up"]},{"osd":1,"uuid":"ade3b803-599d-48f1-9ef3-4ace0e8a41f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":19886},{"type":"v1","addr":"172.21.15.116:6817","nonce":19886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":19886},{"type":"v1","addr":"172.21.15.116:6819","nonce":19886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":19886},{"type":"v1","addr":"172.21.15.116:6823","nonce":19886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":19886},{"type":"v1","addr":"172.21.15.116:6821","nonce":19886}]},"public_addr":"172.21.15.116:6817/19886","cluster_addr":"172.21.15.116:6819/19886","heartbeat_back_addr":"172.21.15.116:6823/19886","heartbeat_front_addr":"172.21.15.116:6821/19886","state":["exists","up"]},{"osd":2,"uuid":"03cf29d4-35d2-4971-a6a5-9cdfef6d93b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":19884},{"type":"v1","addr":"172.21.15.116:6801","nonce":19884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":19884},{"type":"v1","addr":"172.21.15.116:6803","nonce":19884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":19884},{"type":"v1","addr":"172.21.15.116:6807","nonce":19884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":19884},{"type":"v1","addr":"172.21.15.116:6805","nonce":19884}]},"public_addr":"172.21.15.116:6801/19884","cluster_addr":"172.21.15.116:6803/19884","heartbeat_back_addr":"172.21.15.116:6807/19884","heartbeat_front_addr":"172.21.15.116:6805/19884","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.478179+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.449390+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.480145+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:58:50.931 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-21T21:58:50.931 INFO:tasks.ceph:Creating RBD pool 2024-08-21T21:58:50.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster c1 osd pool create rbd 8 2024-08-21T21:58:51.393 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'rbd' created 2024-08-21T21:58:51.407 DEBUG:teuthology.orchestra.run.smithi116:> rbd --cluster c1 pool init rbd 2024-08-21T21:58:51.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:58:51.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:58:54.428 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2024-08-21T21:58:54.428 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2024-08-21T21:58:54.428 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2024-08-21T21:58:54.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2024-08-21T21:58:54.429 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-21T21:58:54.781 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:54.792 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":19548},{"type":"v1","addr":"172.21.15.116:6825","nonce":19548}]},"active_addr":"172.21.15.116:6825/19548","active_change":"2024-08-21T21:58:40.111297+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2552713954}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":4179360893}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1346223842}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":3279382248}]}]} 2024-08-21T21:58:54.794 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2024-08-21T21:58:54.794 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2024-08-21T21:58:54.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-08-21T21:58:55.050 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:55.051 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"fsid":"e8a4fd3f-20d2-4a73-ba88-4d584db938fc","created":"2024-08-21T21:58:36.273783+0000","modified":"2024-08-21T21:58:54.399447+0000","last_up_change":"2024-08-21T21:58:44.377019+0000","last_in_change":"2024-08-21T21:58:37.092938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-21T21:58:46.142988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-21T21:58:51.202772+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7ade91d4-cc6f-46b1-b954-63937c222eda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":19885},{"type":"v1","addr":"172.21.15.116:6809","nonce":19885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":19885},{"type":"v1","addr":"172.21.15.116:6811","nonce":19885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":19885},{"type":"v1","addr":"172.21.15.116:6815","nonce":19885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":19885},{"type":"v1","addr":"172.21.15.116:6813","nonce":19885}]},"public_addr":"172.21.15.116:6809/19885","cluster_addr":"172.21.15.116:6811/19885","heartbeat_back_addr":"172.21.15.116:6815/19885","heartbeat_front_addr":"172.21.15.116:6813/19885","state":["exists","up"]},{"osd":1,"uuid":"ade3b803-599d-48f1-9ef3-4ace0e8a41f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":19886},{"type":"v1","addr":"172.21.15.116:6817","nonce":19886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":19886},{"type":"v1","addr":"172.21.15.116:6819","nonce":19886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":19886},{"type":"v1","addr":"172.21.15.116:6823","nonce":19886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":19886},{"type":"v1","addr":"172.21.15.116:6821","nonce":19886}]},"public_addr":"172.21.15.116:6817/19886","cluster_addr":"172.21.15.116:6819/19886","heartbeat_back_addr":"172.21.15.116:6823/19886","heartbeat_front_addr":"172.21.15.116:6821/19886","state":["exists","up"]},{"osd":2,"uuid":"03cf29d4-35d2-4971-a6a5-9cdfef6d93b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":19884},{"type":"v1","addr":"172.21.15.116:6801","nonce":19884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":19884},{"type":"v1","addr":"172.21.15.116:6803","nonce":19884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":19884},{"type":"v1","addr":"172.21.15.116:6807","nonce":19884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":19884},{"type":"v1","addr":"172.21.15.116:6805","nonce":19884}]},"public_addr":"172.21.15.116:6801/19884","cluster_addr":"172.21.15.116:6803/19884","heartbeat_back_addr":"172.21.15.116:6807/19884","heartbeat_front_addr":"172.21.15.116:6805/19884","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.478179+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.449390+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.480145+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:58:55.063 INFO:tasks.ceph.ceph_manager.c1:all up! 2024-08-21T21:58:55.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-08-21T21:58:55.321 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:55.321 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"fsid":"e8a4fd3f-20d2-4a73-ba88-4d584db938fc","created":"2024-08-21T21:58:36.273783+0000","modified":"2024-08-21T21:58:54.399447+0000","last_up_change":"2024-08-21T21:58:44.377019+0000","last_in_change":"2024-08-21T21:58:37.092938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-21T21:58:46.142988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-21T21:58:51.202772+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7ade91d4-cc6f-46b1-b954-63937c222eda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":19885},{"type":"v1","addr":"172.21.15.116:6809","nonce":19885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":19885},{"type":"v1","addr":"172.21.15.116:6811","nonce":19885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":19885},{"type":"v1","addr":"172.21.15.116:6815","nonce":19885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":19885},{"type":"v1","addr":"172.21.15.116:6813","nonce":19885}]},"public_addr":"172.21.15.116:6809/19885","cluster_addr":"172.21.15.116:6811/19885","heartbeat_back_addr":"172.21.15.116:6815/19885","heartbeat_front_addr":"172.21.15.116:6813/19885","state":["exists","up"]},{"osd":1,"uuid":"ade3b803-599d-48f1-9ef3-4ace0e8a41f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":19886},{"type":"v1","addr":"172.21.15.116:6817","nonce":19886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":19886},{"type":"v1","addr":"172.21.15.116:6819","nonce":19886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":19886},{"type":"v1","addr":"172.21.15.116:6823","nonce":19886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":19886},{"type":"v1","addr":"172.21.15.116:6821","nonce":19886}]},"public_addr":"172.21.15.116:6817/19886","cluster_addr":"172.21.15.116:6819/19886","heartbeat_back_addr":"172.21.15.116:6823/19886","heartbeat_front_addr":"172.21.15.116:6821/19886","state":["exists","up"]},{"osd":2,"uuid":"03cf29d4-35d2-4971-a6a5-9cdfef6d93b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":19884},{"type":"v1","addr":"172.21.15.116:6801","nonce":19884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":19884},{"type":"v1","addr":"172.21.15.116:6803","nonce":19884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":19884},{"type":"v1","addr":"172.21.15.116:6807","nonce":19884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":19884},{"type":"v1","addr":"172.21.15.116:6805","nonce":19884}]},"public_addr":"172.21.15.116:6801/19884","cluster_addr":"172.21.15.116:6803/19884","heartbeat_back_addr":"172.21.15.116:6807/19884","heartbeat_front_addr":"172.21.15.116:6805/19884","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.478179+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.449390+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:58:41.480145+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:58:55.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2024-08-21T21:58:55.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2024-08-21T21:58:55.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2024-08-21T21:58:55.464 INFO:teuthology.orchestra.run.smithi116.stdout:34359738374 2024-08-21T21:58:55.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-08-21T21:58:55.511 INFO:teuthology.orchestra.run.smithi116.stdout:34359738375 2024-08-21T21:58:55.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-08-21T21:58:55.512 INFO:teuthology.orchestra.run.smithi116.stdout:34359738373 2024-08-21T21:58:55.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-08-21T21:58:55.822 INFO:teuthology.orchestra.run.smithi116.stdout:34359738372 2024-08-21T21:58:55.830 INFO:teuthology.orchestra.run.smithi116.stdout:34359738371 2024-08-21T21:58:55.835 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738372 for osd.0 2024-08-21T21:58:55.836 INFO:teuthology.orchestra.run.smithi116.stdout:34359738373 2024-08-21T21:58:55.843 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738371 for osd.2 2024-08-21T21:58:55.848 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738375 got 34359738373 for osd.1 2024-08-21T21:58:56.837 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2024-08-21T21:58:56.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2024-08-21T21:58:56.849 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2024-08-21T21:58:57.131 INFO:teuthology.orchestra.run.smithi116.stdout:34359738374 2024-08-21T21:58:57.144 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738374 got 34359738374 for osd.0 2024-08-21T21:58:57.144 DEBUG:teuthology.parallel:result is None 2024-08-21T21:58:57.164 INFO:teuthology.orchestra.run.smithi116.stdout:34359738375 2024-08-21T21:58:57.169 INFO:teuthology.orchestra.run.smithi116.stdout:34359738373 2024-08-21T21:58:57.177 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738375 got 34359738375 for osd.1 2024-08-21T21:58:57.177 DEBUG:teuthology.parallel:result is None 2024-08-21T21:58:57.182 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.2 2024-08-21T21:58:57.182 DEBUG:teuthology.parallel:result is None 2024-08-21T21:58:57.182 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2024-08-21T21:58:57.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-08-21T21:58:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:57.481 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-21T21:58:57.492 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-08-21T21:58:56.117517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893640,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":313679160,"statfs":{"total":322122547200,"available":321207459840,"internally_reserved":0,"allocated":1605632,"data_stored":1298916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.725300"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.411920+0000","last_change":"2024-08-21T21:58:54.412136+0000","last_active":"2024-08-21T21:58:54.411920+0000","last_peered":"2024-08-21T21:58:54.411920+0000","last_clean":"2024-08-21T21:58:54.411920+0000","last_became_active":"2024-08-21T21:58:52.406134+0000","last_became_peered":"2024-08-21T21:58:52.406134+0000","last_unstale":"2024-08-21T21:58:54.411920+0000","last_undegraded":"2024-08-21T21:58:54.411920+0000","last_fullsized":"2024-08-21T21:58:54.411920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T06:10:12.565667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040101999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.412473+0000","last_change":"2024-08-21T21:58:54.412689+0000","last_active":"2024-08-21T21:58:54.412473+0000","last_peered":"2024-08-21T21:58:54.412473+0000","last_clean":"2024-08-21T21:58:54.412473+0000","last_became_active":"2024-08-21T21:58:52.406034+0000","last_became_peered":"2024-08-21T21:58:52.406034+0000","last_unstale":"2024-08-21T21:58:54.412473+0000","last_undegraded":"2024-08-21T21:58:54.412473+0000","last_fullsized":"2024-08-21T21:58:54.412473+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T00:27:55.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080365199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.411968+0000","last_change":"2024-08-21T21:58:54.412188+0000","last_active":"2024-08-21T21:58:54.411968+0000","last_peered":"2024-08-21T21:58:54.411968+0000","last_clean":"2024-08-21T21:58:54.411968+0000","last_became_active":"2024-08-21T21:58:52.406072+0000","last_became_peered":"2024-08-21T21:58:52.406072+0000","last_unstale":"2024-08-21T21:58:54.411968+0000","last_undegraded":"2024-08-21T21:58:54.411968+0000","last_fullsized":"2024-08-21T21:58:54.411968+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T23:16:04.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042411599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.412205+0000","last_change":"2024-08-21T21:58:54.412380+0000","last_active":"2024-08-21T21:58:54.412205+0000","last_peered":"2024-08-21T21:58:54.412205+0000","last_clean":"2024-08-21T21:58:54.412205+0000","last_became_active":"2024-08-21T21:58:52.406172+0000","last_became_peered":"2024-08-21T21:58:52.406172+0000","last_unstale":"2024-08-21T21:58:54.412205+0000","last_undegraded":"2024-08-21T21:58:54.412205+0000","last_fullsized":"2024-08-21T21:58:54.412205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:10:16.564694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061332100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.409231+0000","last_change":"2024-08-21T21:58:54.409231+0000","last_active":"2024-08-21T21:58:54.409231+0000","last_peered":"2024-08-21T21:58:54.409231+0000","last_clean":"2024-08-21T21:58:54.409231+0000","last_became_active":"2024-08-21T21:58:52.406027+0000","last_became_peered":"2024-08-21T21:58:52.406027+0000","last_unstale":"2024-08-21T21:58:54.409231+0000","last_undegraded":"2024-08-21T21:58:54.409231+0000","last_fullsized":"2024-08-21T21:58:54.409231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:21:00.789826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000307218,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:55.237398+0000","last_change":"2024-08-21T21:58:55.237525+0000","last_active":"2024-08-21T21:58:55.237398+0000","last_peered":"2024-08-21T21:58:55.237398+0000","last_clean":"2024-08-21T21:58:55.237398+0000","last_became_active":"2024-08-21T21:58:52.403274+0000","last_became_peered":"2024-08-21T21:58:52.403274+0000","last_unstale":"2024-08-21T21:58:55.237398+0000","last_undegraded":"2024-08-21T21:58:55.237398+0000","last_fullsized":"2024-08-21T21:58:55.237398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T07:39:24.639235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000213341,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:55.237436+0000","last_change":"2024-08-21T21:58:55.237555+0000","last_active":"2024-08-21T21:58:55.237436+0000","last_peered":"2024-08-21T21:58:55.237436+0000","last_clean":"2024-08-21T21:58:55.237436+0000","last_became_active":"2024-08-21T21:58:52.403229+0000","last_became_peered":"2024-08-21T21:58:52.403229+0000","last_unstale":"2024-08-21T21:58:55.237436+0000","last_undegraded":"2024-08-21T21:58:55.237436+0000","last_fullsized":"2024-08-21T21:58:55.237436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T02:14:51.994816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023340200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.412270+0000","last_change":"2024-08-21T21:58:54.412456+0000","last_active":"2024-08-21T21:58:54.412270+0000","last_peered":"2024-08-21T21:58:54.412270+0000","last_clean":"2024-08-21T21:58:54.412270+0000","last_became_active":"2024-08-21T21:58:52.401081+0000","last_became_peered":"2024-08-21T21:58:52.401081+0000","last_unstale":"2024-08-21T21:58:54.412270+0000","last_undegraded":"2024-08-21T21:58:54.412270+0000","last_fullsized":"2024-08-21T21:58:54.412270+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:24:45.393737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079099300000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-21T21:58:53.401976+0000","last_change":"2024-08-21T21:58:47.400031+0000","last_active":"2024-08-21T21:58:53.401976+0000","last_peered":"2024-08-21T21:58:53.401976+0000","last_clean":"2024-08-21T21:58:53.401976+0000","last_became_active":"2024-08-21T21:58:47.399122+0000","last_became_peered":"2024-08-21T21:58:47.399122+0000","last_unstale":"2024-08-21T21:58:53.401976+0000","last_undegraded":"2024-08-21T21:58:53.401976+0000","last_fullsized":"2024-08-21T21:58:53.401976+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:46.378872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:46.378872+0000","last_clean_scrub_stamp":"2024-08-21T21:58:46.378872+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:57:21.226154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104560116,"statfs":{"total":107374182400,"available":107069558784,"internally_reserved":0,"allocated":131072,"data_stored":37014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.17899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298072,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559528,"statfs":{"total":107374182400,"available":107068956672,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559516,"statfs":{"total":107374182400,"available":107068944384,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-21T21:58:57.493 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2024-08-21T21:58:57.747 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:57.748 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-21T21:58:57.758 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-08-21T21:58:56.117517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893640,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":313679160,"statfs":{"total":322122547200,"available":321207459840,"internally_reserved":0,"allocated":1605632,"data_stored":1298916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.725300"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.411920+0000","last_change":"2024-08-21T21:58:54.412136+0000","last_active":"2024-08-21T21:58:54.411920+0000","last_peered":"2024-08-21T21:58:54.411920+0000","last_clean":"2024-08-21T21:58:54.411920+0000","last_became_active":"2024-08-21T21:58:52.406134+0000","last_became_peered":"2024-08-21T21:58:52.406134+0000","last_unstale":"2024-08-21T21:58:54.411920+0000","last_undegraded":"2024-08-21T21:58:54.411920+0000","last_fullsized":"2024-08-21T21:58:54.411920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T06:10:12.565667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040101999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.412473+0000","last_change":"2024-08-21T21:58:54.412689+0000","last_active":"2024-08-21T21:58:54.412473+0000","last_peered":"2024-08-21T21:58:54.412473+0000","last_clean":"2024-08-21T21:58:54.412473+0000","last_became_active":"2024-08-21T21:58:52.406034+0000","last_became_peered":"2024-08-21T21:58:52.406034+0000","last_unstale":"2024-08-21T21:58:54.412473+0000","last_undegraded":"2024-08-21T21:58:54.412473+0000","last_fullsized":"2024-08-21T21:58:54.412473+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T00:27:55.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080365199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.411968+0000","last_change":"2024-08-21T21:58:54.412188+0000","last_active":"2024-08-21T21:58:54.411968+0000","last_peered":"2024-08-21T21:58:54.411968+0000","last_clean":"2024-08-21T21:58:54.411968+0000","last_became_active":"2024-08-21T21:58:52.406072+0000","last_became_peered":"2024-08-21T21:58:52.406072+0000","last_unstale":"2024-08-21T21:58:54.411968+0000","last_undegraded":"2024-08-21T21:58:54.411968+0000","last_fullsized":"2024-08-21T21:58:54.411968+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T23:16:04.701800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042411599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.412205+0000","last_change":"2024-08-21T21:58:54.412380+0000","last_active":"2024-08-21T21:58:54.412205+0000","last_peered":"2024-08-21T21:58:54.412205+0000","last_clean":"2024-08-21T21:58:54.412205+0000","last_became_active":"2024-08-21T21:58:52.406172+0000","last_became_peered":"2024-08-21T21:58:52.406172+0000","last_unstale":"2024-08-21T21:58:54.412205+0000","last_undegraded":"2024-08-21T21:58:54.412205+0000","last_fullsized":"2024-08-21T21:58:54.412205+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:10:16.564694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061332100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.409231+0000","last_change":"2024-08-21T21:58:54.409231+0000","last_active":"2024-08-21T21:58:54.409231+0000","last_peered":"2024-08-21T21:58:54.409231+0000","last_clean":"2024-08-21T21:58:54.409231+0000","last_became_active":"2024-08-21T21:58:52.406027+0000","last_became_peered":"2024-08-21T21:58:52.406027+0000","last_unstale":"2024-08-21T21:58:54.409231+0000","last_undegraded":"2024-08-21T21:58:54.409231+0000","last_fullsized":"2024-08-21T21:58:54.409231+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:21:00.789826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000307218,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:55.237398+0000","last_change":"2024-08-21T21:58:55.237525+0000","last_active":"2024-08-21T21:58:55.237398+0000","last_peered":"2024-08-21T21:58:55.237398+0000","last_clean":"2024-08-21T21:58:55.237398+0000","last_became_active":"2024-08-21T21:58:52.403274+0000","last_became_peered":"2024-08-21T21:58:52.403274+0000","last_unstale":"2024-08-21T21:58:55.237398+0000","last_undegraded":"2024-08-21T21:58:55.237398+0000","last_fullsized":"2024-08-21T21:58:55.237398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T07:39:24.639235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000213341,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:55.237436+0000","last_change":"2024-08-21T21:58:55.237555+0000","last_active":"2024-08-21T21:58:55.237436+0000","last_peered":"2024-08-21T21:58:55.237436+0000","last_clean":"2024-08-21T21:58:55.237436+0000","last_became_active":"2024-08-21T21:58:52.403229+0000","last_became_peered":"2024-08-21T21:58:52.403229+0000","last_unstale":"2024-08-21T21:58:55.237436+0000","last_undegraded":"2024-08-21T21:58:55.237436+0000","last_fullsized":"2024-08-21T21:58:55.237436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T02:14:51.994816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023340200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-08-21T21:58:54.412270+0000","last_change":"2024-08-21T21:58:54.412456+0000","last_active":"2024-08-21T21:58:54.412270+0000","last_peered":"2024-08-21T21:58:54.412270+0000","last_clean":"2024-08-21T21:58:54.412270+0000","last_became_active":"2024-08-21T21:58:52.401081+0000","last_became_peered":"2024-08-21T21:58:52.401081+0000","last_unstale":"2024-08-21T21:58:54.412270+0000","last_undegraded":"2024-08-21T21:58:54.412270+0000","last_fullsized":"2024-08-21T21:58:54.412270+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:51.388888+0000","last_clean_scrub_stamp":"2024-08-21T21:58:51.388888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:24:45.393737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079099300000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-21T21:58:53.401976+0000","last_change":"2024-08-21T21:58:47.400031+0000","last_active":"2024-08-21T21:58:53.401976+0000","last_peered":"2024-08-21T21:58:53.401976+0000","last_clean":"2024-08-21T21:58:53.401976+0000","last_became_active":"2024-08-21T21:58:47.399122+0000","last_became_peered":"2024-08-21T21:58:47.399122+0000","last_unstale":"2024-08-21T21:58:53.401976+0000","last_undegraded":"2024-08-21T21:58:53.401976+0000","last_fullsized":"2024-08-21T21:58:53.401976+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:58:46.378872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:58:46.378872+0000","last_clean_scrub_stamp":"2024-08-21T21:58:46.378872+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T22:57:21.226154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104560116,"statfs":{"total":107374182400,"available":107069558784,"internally_reserved":0,"allocated":131072,"data_stored":37014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.17899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298072,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559528,"statfs":{"total":107374182400,"available":107068956672,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559516,"statfs":{"total":107374182400,"available":107068944384,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-21T21:58:57.759 INFO:tasks.ceph.ceph_manager.c1:clean! 2024-08-21T21:58:57.759 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2024-08-21T21:58:57.760 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2024-08-21T21:58:57.760 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2024-08-21T21:58:58.049 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:58.049 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-21T21:58:58.061 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2024-08-21T21:58:58.061 INFO:teuthology.run_tasks:Running task ceph... 2024-08-21T21:58:58.070 INFO:tasks.ceph:Creating ceph cluster c2... 2024-08-21T21:58:58.070 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-21T21:58:58.071 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274', 'branch': 'wip-yuri7-testing-2024-08-20-1105-quincy', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7866274', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2024-08-20-1105-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri7-testing-2024-08-20-1105-quincy', 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 6565, 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri7-testing-2024-08-20-1105-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMZw3gcuLo5X2pqLvNADs7gKZN4so6LJethSQlhnXNg1d9qR1MWdWYmyGuH5Ktr0PrbjgDDhWg5WK2aczrvW/zA=', 'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBwiuhM1fPYG8eHoRh/WKgq0Bvla0NGDeBYIL52Cb/n8CRt+ckwfBetifQpkEOPmoN0phJKjpnMu2pcB0AwETrs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b5e746614a5ba6992e5be76bf3df447c931f3748', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '4d4c5e30374204f3b32f867e61bb4e1a3880f672', 'timestamp': '2024-08-21_20:32:34', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-21T21:58:58.071 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-08-21T21:58:58.074 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2024-08-21T21:58:58.093 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-08-21T21:58:58.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2024-08-21T21:58:58.161 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:58.162 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-08-21T21:58:58.219 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-21T21:58:58.219 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-08-21T21:58:58.275 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-08-21 21:58:14.657731206 +0000 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-08-21 21:55:54.372150578 +0000 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-08-21 21:55:54.372150578 +0000 2024-08-21T21:58:58.276 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-08-21 21:55:54.372150578 +0000 2024-08-21T21:58:58.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-21T21:58:58.339 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-08-21T21:58:58.339 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-08-21T21:58:58.339 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000102384 s, 5.0 MB/s 2024-08-21T21:58:58.340 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-21T21:58:58.397 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-08-21T21:58:58.453 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-21T21:58:58.453 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:58.453 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-21T21:58:58.453 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:58.453 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:58.453 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-08-21 21:58:14.658731187 +0000 2024-08-21T21:58:58.453 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-08-21 21:55:54.684144644 +0000 2024-08-21T21:58:58.454 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-08-21 21:55:54.684144644 +0000 2024-08-21T21:58:58.454 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-08-21 21:55:54.684144644 +0000 2024-08-21T21:58:58.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-21T21:58:58.517 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-08-21T21:58:58.517 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-08-21T21:58:58.517 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.204e-05 s, 6.2 MB/s 2024-08-21T21:58:58.518 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-21T21:58:58.575 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-08-21T21:58:58.630 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-08-21 21:58:14.658731187 +0000 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-08-21 21:55:55.003138576 +0000 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-08-21 21:55:55.003138576 +0000 2024-08-21T21:58:58.631 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-08-21 21:55:55.003138576 +0000 2024-08-21T21:58:58.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-21T21:58:58.694 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-08-21T21:58:58.694 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-08-21T21:58:58.694 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.1231e-05 s, 5.6 MB/s 2024-08-21T21:58:58.695 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-21T21:58:58.751 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-08-21T21:58:58.806 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-21T21:58:58.806 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-21T21:58:58.807 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-08-21T21:58:58.807 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-21T21:58:58.807 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-08-21T21:58:58.807 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-08-21 21:58:14.658731187 +0000 2024-08-21T21:58:58.807 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-08-21 21:55:55.307132794 +0000 2024-08-21T21:58:58.807 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-08-21 21:55:55.307132794 +0000 2024-08-21T21:58:58.807 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-08-21 21:55:55.307132794 +0000 2024-08-21T21:58:58.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-21T21:58:58.874 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-08-21T21:58:58.875 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-08-21T21:58:58.875 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000102961 s, 5.0 MB/s 2024-08-21T21:58:58.875 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-21T21:58:58.930 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:58:58.930 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-21T21:58:58.930 INFO:tasks.ceph:Generating config... 2024-08-21T21:58:58.931 INFO:tasks.ceph:[client] debug rgw = 20 2024-08-21T21:58:58.931 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw data log num shards = 4 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw md log max shards = 4 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] setgroup = ceph 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client] setuser = ceph 2024-08-21T21:58:58.932 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2024-08-21T21:58:58.932 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-21T21:58:58.932 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-21T21:58:58.932 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-21T21:58:58.933 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-21T21:58:58.933 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-21T21:58:58.933 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-21T21:58:58.933 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-21T21:58:58.933 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2024-08-21T21:58:58.933 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2024-08-21T21:58:58.933 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2024-08-21T21:58:58.933 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2024-08-21T21:58:58.933 INFO:tasks.ceph:Setting up c2.mon.a... 2024-08-21T21:58:58.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2024-08-21T21:58:59.014 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/c2.keyring 2024-08-21T21:58:59.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2024-08-21T21:58:59.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-08-21T21:58:59.160 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.178')] 2024-08-21T21:58:59.160 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.178'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2024-08-21T21:58:59.162 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:59.162 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-21T21:58:59.215 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.178 --print /home/ubuntu/cephtest/c2.monmap 2024-08-21T21:58:59.290 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:58:59.290 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:monmaptool: generated fsid 963999ae-000a-4464-a440-8bde81eb6b50 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:setting min_mon_release = octopus 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:epoch 0 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:fsid 963999ae-000a-4464-a440-8bde81eb6b50 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:last_changed 2024-08-21T21:58:59.290264+0000 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:created 2024-08-21T21:58:59.290264+0000 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:min_mon_release 15 (octopus) 2024-08-21T21:58:59.291 INFO:teuthology.orchestra.run.smithi178.stdout:election_strategy: 1 2024-08-21T21:58:59.292 INFO:teuthology.orchestra.run.smithi178.stdout:0: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.a 2024-08-21T21:58:59.292 INFO:teuthology.orchestra.run.smithi178.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2024-08-21T21:58:59.293 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-21T21:58:59.347 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 963999ae-000a-4464-a440-8bde81eb6b50... 2024-08-21T21:58:59.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-08-21T21:58:59.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2024-08-21T21:58:59.399 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-08-21T21:58:59.399 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-08-21T21:58:59.399 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.400 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-08-21T21:58:59.401 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-08-21T21:58:59.402 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = 172.21.15.178 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 963999ae-000a-4464-a440-8bde81eb6b50 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-08-21T21:58:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.404 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: rocksdb delete range threshold = 0 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-08-21T21:58:59.405 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-21T21:58:59.406 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-08-21T21:58:59.407 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: debug rgw = 20 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt require ssl = False 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt s3 kms backend = testing 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw curl low speed time = 300 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw data log num shards = 4 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw md log max shards = 4 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw sync log trim interval = 0 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw sync meta inject err probability = 0.1 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: rgw sync obj etag verify = True 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: setgroup = ceph 2024-08-21T21:58:59.408 INFO:teuthology.orchestra.run.smithi116.stdout: setuser = ceph 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout: rgw data notify interval msec = 0 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout: osd_max_omap_entries_per_request = 10 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout: osd_max_omap_entries_per_request = 1000 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-08-21T21:58:59.409 INFO:teuthology.orchestra.run.smithi116.stdout: osd_max_omap_entries_per_request = 10000 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-08-21T21:58:59.431 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-08-21T21:58:59.432 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.433 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = 172.21.15.178 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = 963999ae-000a-4464-a440-8bde81eb6b50 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-08-21T21:58:59.434 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.435 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: rocksdb delete range threshold = 0 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-08-21T21:58:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-08-21T21:58:59.437 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-08-21T21:58:59.438 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: debug rgw = 20 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: rgw crypt require ssl = False 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: rgw crypt s3 kms backend = testing 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2024-08-21T21:58:59.439 INFO:teuthology.orchestra.run.smithi178.stdout: rgw curl low speed time = 300 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: rgw data log num shards = 4 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: rgw md log max shards = 4 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: rgw sync log trim interval = 0 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: rgw sync meta inject err probability = 0.1 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: rgw sync obj etag verify = True 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: setgroup = ceph 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: setuser = ceph 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: rgw data notify interval msec = 0 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: osd_max_omap_entries_per_request = 10 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout: osd_max_omap_entries_per_request = 1000 2024-08-21T21:58:59.440 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-08-21T21:58:59.441 INFO:teuthology.orchestra.run.smithi178.stdout: osd_max_omap_entries_per_request = 10000 2024-08-21T21:58:59.442 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2024-08-21T21:58:59.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2024-08-21T21:58:59.535 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-21T21:58:59.535 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:59.535 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2024-08-21T21:58:59.551 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:59.551 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2024-08-21T21:58:59.606 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T21:58:59.607 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:59.607 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/c2.keyring 2024-08-21T21:58:59.607 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-08-21T21:58:59.646 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-21T21:58:59.646 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-08-21T21:58:59.707 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T21:58:59.707 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:59.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/c2.keyring 2024-08-21T21:58:59.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/c2.keyring 2024-08-21T21:58:59.741 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:58:59.742 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/c2.monmap 2024-08-21T21:58:59.797 INFO:tasks.ceph:Setting up mon nodes... 2024-08-21T21:58:59.797 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-21T21:58:59.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2024-08-21T21:58:59.884 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2024-08-21T21:58:59.886 INFO:tasks.ceph:Setting up mds nodes... 2024-08-21T21:58:59.886 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-21T21:58:59.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2024-08-21T21:58:59.923 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/c2.client.0.keyring 2024-08-21T21:58:59.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2024-08-21T21:59:00.013 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/c2.client.1.keyring 2024-08-21T21:59:00.026 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-21T21:59:00.026 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-08-21T21:59:00.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2024-08-21T21:59:00.090 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:59:00.090 INFO:tasks.ceph:role: c2.osd.0 2024-08-21T21:59:00.090 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T21:59:00.090 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-21T21:59:00.155 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-21T21:59:00.156 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-21T21:59:00.186 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-08-21T21:59:00.188 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-08-21T21:59:00.189 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2024-08-21T21:59:00.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2024-08-21T21:59:00.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2024-08-21T21:59:00.393 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:59:00.393 INFO:tasks.ceph:role: c2.osd.1 2024-08-21T21:59:00.393 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T21:59:00.394 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-21T21:59:00.458 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-21T21:59:00.490 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-08-21T21:59:00.492 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-08-21T21:59:00.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2024-08-21T21:59:00.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2024-08-21T21:59:00.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2024-08-21T21:59:00.695 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2024-08-21T21:59:00.696 INFO:tasks.ceph:role: c2.osd.2 2024-08-21T21:59:00.696 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T21:59:00.696 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-21T21:59:00.760 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-21T21:59:00.760 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-21T21:59:00.760 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-21T21:59:00.760 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-21T21:59:00.760 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-21T21:59:00.760 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-08-21T21:59:00.760 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-21T21:59:00.761 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-21T21:59:00.761 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-21T21:59:00.761 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-21T21:59:00.793 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-08-21T21:59:00.795 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-08-21T21:59:00.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2024-08-21T21:59:00.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2024-08-21T21:59:00.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2024-08-21T21:59:01.037 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:01.036+0000 7fbf1c833440 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2024-08-21T21:59:01.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:01.036+0000 7fbf1c833440 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2024-08-21T21:59:01.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:01.036+0000 7fbf1c833440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-08-21T21:59:01.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:01.036+0000 7fbf1c833440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-08-21T21:59:01.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:01.036+0000 7fbf1c833440 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2024-08-21T21:59:01.038 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:01.036+0000 7fbf1c833440 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2024-08-21T21:59:01.559 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:01.558+0000 7fbf1c833440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:59:02.320 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:02.319+0000 7fbf1c833440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:59:03.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2024-08-21T21:59:03.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2024-08-21T21:59:03.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:03.219+0000 7f35ff68b440 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2024-08-21T21:59:03.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:03.219+0000 7f35ff68b440 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2024-08-21T21:59:03.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:03.219+0000 7f35ff68b440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-08-21T21:59:03.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:03.219+0000 7f35ff68b440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-08-21T21:59:03.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:03.219+0000 7f35ff68b440 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2024-08-21T21:59:03.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:03.220+0000 7f35ff68b440 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2024-08-21T21:59:03.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:03.742+0000 7f35ff68b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:59:04.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:04.504+0000 7f35ff68b440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:59:05.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2024-08-21T21:59:05.321 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2024-08-21T21:59:05.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:05.405+0000 7ffbcb60f440 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2024-08-21T21:59:05.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:05.405+0000 7ffbcb60f440 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2024-08-21T21:59:05.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:05.405+0000 7ffbcb60f440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-08-21T21:59:05.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:05.405+0000 7ffbcb60f440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-08-21T21:59:05.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:05.405+0000 7ffbcb60f440 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2024-08-21T21:59:05.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:05.405+0000 7ffbcb60f440 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2024-08-21T21:59:05.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:05.928+0000 7ffbcb60f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:59:06.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:06.693+0000 7ffbcb60f440 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-08-21T21:59:07.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2024-08-21T21:59:07.517 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-21T21:59:07.517 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:07.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2024-08-21T21:59:07.585 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:07.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2024-08-21T21:59:07.652 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:07.652 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2024-08-21T21:59:07.718 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:07.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2024-08-21T21:59:07.784 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:07.784 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2024-08-21T21:59:07.839 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:07.840 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2024-08-21T21:59:07.895 INFO:tasks.ceph:Adding keys to all mons... 2024-08-21T21:59:07.895 DEBUG:teuthology.orchestra.run.smithi178:> sudo tee -a /etc/ceph/c2.keyring 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.x] 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAjY8ZmTYauNBAAy93s+gVlCvzPRqO/GBbCAA== 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAlY8Zm2As0AhAAv0++c63ydpV13vskaFICzw== 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAnY8ZmZVkjDRAAWN03aTJlwzDgzLB1JJKpWQ== 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQApY8Zmnc03GBAA3dGerD7NTJfL1BR4pzGqdA== 2024-08-21T21:59:07.958 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-08-21T21:59:07.959 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAjY8ZmE+oDNxAA6SDJtiGCwCPkJfYPM+dGyQ== 2024-08-21T21:59:07.959 INFO:teuthology.orchestra.run.smithi178.stdout:[client.1] 2024-08-21T21:59:07.959 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQAkY8ZmzJ/BABAAqnk2YtwYDdEd4p99swAdXg== 2024-08-21T21:59:07.959 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-21T21:59:08.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-21T21:59:08.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-21T21:59:08.138 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-21T21:59:08.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-21T21:59:08.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-21T21:59:08.282 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-21T21:59:08.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2024-08-21T21:59:08.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2024-08-21T21:59:08.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2024-08-21T21:59:08.423 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/c2.monmap 2024-08-21T21:59:08.478 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2024-08-21T21:59:08.479 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2024-08-21T21:59:08.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2024-08-21T21:59:08.521 INFO:tasks.ceph.c2.mon.a:Started 2024-08-21T21:59:08.521 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2024-08-21T21:59:08.521 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2024-08-21T21:59:08.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2024-08-21T21:59:08.523 INFO:tasks.ceph.c2.mgr.x:Started 2024-08-21T21:59:08.523 INFO:tasks.ceph:Setting crush tunables to default 2024-08-21T21:59:08.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster c2 osd crush tunables default 2024-08-21T21:59:08.686 INFO:teuthology.orchestra.run.smithi178.stderr:adjusted tunables profile to default 2024-08-21T21:59:08.696 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2024-08-21T21:59:08.696 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:08.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2024-08-21T21:59:08.721 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:08.721 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2024-08-21T21:59:08.785 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-08-21T21:59:08.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2024-08-21T21:59:08.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster c2 osd new d9652f75-1900-4c11-9db5-f3adedac6cf4 0 2024-08-21T21:59:09.073 INFO:teuthology.orchestra.run.smithi178.stdout:0 2024-08-21T21:59:09.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster c2 osd new 929d5722-725a-4d2c-a9fc-563930bf59d3 1 2024-08-21T21:59:09.289 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-08-21T21:59:09.300 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster c2 osd new 14fd26b5-9422-4d68-bb43-ce789449b6e4 2 2024-08-21T21:59:09.401 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:09.400+0000 7fa34f028ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-21T21:59:09.470 INFO:teuthology.orchestra.run.smithi178.stdout:2 2024-08-21T21:59:09.482 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2024-08-21T21:59:09.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2024-08-21T21:59:09.484 INFO:tasks.ceph.c2.osd.0:Started 2024-08-21T21:59:09.484 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2024-08-21T21:59:09.484 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2024-08-21T21:59:09.486 INFO:tasks.ceph.c2.osd.1:Started 2024-08-21T21:59:09.487 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2024-08-21T21:59:09.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2024-08-21T21:59:09.489 INFO:tasks.ceph.c2.osd.2:Started 2024-08-21T21:59:09.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-08-21T21:59:09.540 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:09.538+0000 7fa34f028ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-21T21:59:09.657 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:59:09.657 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":5,"fsid":"963999ae-000a-4464-a440-8bde81eb6b50","created":"2024-08-21T21:59:08.600909+0000","modified":"2024-08-21T21:59:09.468854+0000","last_up_change":"0.000000","last_in_change":"2024-08-21T21:59:09.468854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d9652f75-1900-4c11-9db5-f3adedac6cf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"929d5722-725a-4d2c-a9fc-563930bf59d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"14fd26b5-9422-4d68-bb43-ce789449b6e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:59:09.668 INFO:tasks.ceph.ceph_manager.c2:[] 2024-08-21T21:59:09.669 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-21T21:59:09.689 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:09.687+0000 7fa34f028ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-21T21:59:09.778 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:09.776+0000 7fa34f028ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-21T21:59:09.925 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:09.923+0000 7fa34f028ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-21T21:59:09.971 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-08-21T21:59:10.033 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.031+0000 7fa34f028ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-21T21:59:10.081 INFO:tasks.ceph.c2.osd.1.smithi178.stderr:2024-08-21T21:59:10.079+0000 7fd6fa70f440 -1 Falling back to public interface 2024-08-21T21:59:10.083 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.081+0000 7fa34f028ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-21T21:59:10.086 INFO:tasks.ceph.c2.osd.2.smithi178.stderr:2024-08-21T21:59:10.085+0000 7f1472553440 -1 Falling back to public interface 2024-08-21T21:59:10.097 INFO:tasks.ceph.c2.osd.0.smithi178.stderr:2024-08-21T21:59:10.095+0000 7f9e64d21440 -1 Falling back to public interface 2024-08-21T21:59:10.122 INFO:teuthology.misc.health.smithi178.stdout: 2024-08-21T21:59:10.122 INFO:teuthology.misc.health.smithi178.stdout:{"epoch":5,"fsid":"963999ae-000a-4464-a440-8bde81eb6b50","created":"2024-08-21T21:59:08.600909+0000","modified":"2024-08-21T21:59:09.468854+0000","last_up_change":"0.000000","last_in_change":"2024-08-21T21:59:09.468854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d9652f75-1900-4c11-9db5-f3adedac6cf4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"929d5722-725a-4d2c-a9fc-563930bf59d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"14fd26b5-9422-4d68-bb43-ce789449b6e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:59:10.133 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-08-21T21:59:10.154 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.153+0000 7fa34f028ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-21T21:59:10.243 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.242+0000 7fa34f028ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-21T21:59:10.282 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.281+0000 7fa34f028ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-21T21:59:10.377 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.375+0000 7fa34f028ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-21T21:59:10.420 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.418+0000 7fa34f028ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-21T21:59:10.459 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:10.457+0000 7fa34f028ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-21T21:59:11.532 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:11.530+0000 7fa34f028ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-21T21:59:11.579 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:11.578+0000 7fa34f028ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-21T21:59:11.628 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:11.627+0000 7fa34f028ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-21T21:59:11.679 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:11.677+0000 7fa34f028ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-21T21:59:11.774 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:11.772+0000 7fa34f028ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-21T21:59:11.816 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:11.815+0000 7fa34f028ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-21T21:59:11.902 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:11.900+0000 7fa34f028ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-21T21:59:12.302 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:12.300+0000 7fa34f028ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-21T21:59:12.348 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:12.346+0000 7fa34f028ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-21T21:59:12.479 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:12.477+0000 7fa34f028ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-21T21:59:12.529 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:12.528+0000 7fa34f028ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-21T21:59:12.807 INFO:tasks.ceph.c2.osd.1.smithi178.stderr:2024-08-21T21:59:12.805+0000 7fd6fa70f440 -1 osd.1 0 log_to_monitors true 2024-08-21T21:59:12.828 INFO:tasks.ceph.c2.osd.2.smithi178.stderr:2024-08-21T21:59:12.827+0000 7f1472553440 -1 osd.2 0 log_to_monitors true 2024-08-21T21:59:12.840 INFO:tasks.ceph.c2.osd.0.smithi178.stderr:2024-08-21T21:59:12.839+0000 7f9e64d21440 -1 osd.0 0 log_to_monitors true 2024-08-21T21:59:13.536 INFO:tasks.ceph.c2.mgr.x.smithi178.stderr:2024-08-21T21:59:13.535+0000 7fa333cc6640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-21T21:59:15.295 INFO:tasks.ceph.c2.osd.0.smithi178.stderr:2024-08-21T21:59:15.293+0000 7f9e5f7f6640 -1 osd.0 0 waiting for initial osdmap 2024-08-21T21:59:15.298 INFO:tasks.ceph.c2.osd.0.smithi178.stderr:2024-08-21T21:59:15.296+0000 7f9e587e8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-21T21:59:15.311 INFO:tasks.ceph.c2.osd.2.smithi178.stderr:2024-08-21T21:59:15.309+0000 7f146cff6640 -1 osd.2 0 waiting for initial osdmap 2024-08-21T21:59:15.313 INFO:tasks.ceph.c2.osd.2.smithi178.stderr:2024-08-21T21:59:15.312+0000 7f1465fe8640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-21T21:59:15.336 INFO:tasks.ceph.c2.osd.1.smithi178.stderr:2024-08-21T21:59:15.335+0000 7fd6f57f6640 -1 osd.1 0 waiting for initial osdmap 2024-08-21T21:59:15.338 INFO:tasks.ceph.c2.osd.1.smithi178.stderr:2024-08-21T21:59:15.337+0000 7fd6ee7e8640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-21T21:59:16.437 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2024-08-21T21:59:16.704 INFO:teuthology.misc.health.smithi178.stdout: 2024-08-21T21:59:16.705 INFO:teuthology.misc.health.smithi178.stdout:{"epoch":8,"fsid":"963999ae-000a-4464-a440-8bde81eb6b50","created":"2024-08-21T21:59:08.600909+0000","modified":"2024-08-21T21:59:16.295383+0000","last_up_change":"2024-08-21T21:59:16.295383+0000","last_in_change":"2024-08-21T21:59:09.468854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d9652f75-1900-4c11-9db5-f3adedac6cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20064},{"type":"v1","addr":"172.21.15.178:6817","nonce":20064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20064},{"type":"v1","addr":"172.21.15.178:6819","nonce":20064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20064},{"type":"v1","addr":"172.21.15.178:6823","nonce":20064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20064},{"type":"v1","addr":"172.21.15.178:6821","nonce":20064}]},"public_addr":"172.21.15.178:6817/20064","cluster_addr":"172.21.15.178:6819/20064","heartbeat_back_addr":"172.21.15.178:6823/20064","heartbeat_front_addr":"172.21.15.178:6821/20064","state":["exists","up"]},{"osd":1,"uuid":"929d5722-725a-4d2c-a9fc-563930bf59d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20054},{"type":"v1","addr":"172.21.15.178:6801","nonce":20054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20054},{"type":"v1","addr":"172.21.15.178:6803","nonce":20054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":20054},{"type":"v1","addr":"172.21.15.178:6807","nonce":20054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20054},{"type":"v1","addr":"172.21.15.178:6805","nonce":20054}]},"public_addr":"172.21.15.178:6801/20054","cluster_addr":"172.21.15.178:6803/20054","heartbeat_back_addr":"172.21.15.178:6807/20054","heartbeat_front_addr":"172.21.15.178:6805/20054","state":["exists","up"]},{"osd":2,"uuid":"14fd26b5-9422-4d68-bb43-ce789449b6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20055},{"type":"v1","addr":"172.21.15.178:6809","nonce":20055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":20055},{"type":"v1","addr":"172.21.15.178:6811","nonce":20055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20055},{"type":"v1","addr":"172.21.15.178:6815","nonce":20055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20055},{"type":"v1","addr":"172.21.15.178:6813","nonce":20055}]},"public_addr":"172.21.15.178:6809/20055","cluster_addr":"172.21.15.178:6811/20055","heartbeat_back_addr":"172.21.15.178:6815/20055","heartbeat_front_addr":"172.21.15.178:6813/20055","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:59:16.715 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-08-21T21:59:16.715 INFO:tasks.ceph:Creating RBD pool 2024-08-21T21:59:16.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster c2 osd pool create rbd 8 2024-08-21T21:59:17.302 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'rbd' created 2024-08-21T21:59:17.313 DEBUG:teuthology.orchestra.run.smithi178:> rbd --cluster c2 pool init rbd 2024-08-21T21:59:17.344 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:17.344 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:20.327 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2024-08-21T21:59:20.328 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2024-08-21T21:59:20.328 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2024-08-21T21:59:20.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2024-08-21T21:59:20.328 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-21T21:59:20.678 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:59:20.689 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":19718},{"type":"v1","addr":"172.21.15.178:6825","nonce":19718}]},"active_addr":"172.21.15.178:6825/19718","active_change":"2024-08-21T21:59:12.530184+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1958370776}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1402727600}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":2399333242}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":448109470}]}]} 2024-08-21T21:59:20.691 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2024-08-21T21:59:20.691 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2024-08-21T21:59:20.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-08-21T21:59:20.953 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:59:20.953 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":12,"fsid":"963999ae-000a-4464-a440-8bde81eb6b50","created":"2024-08-21T21:59:08.600909+0000","modified":"2024-08-21T21:59:20.300542+0000","last_up_change":"2024-08-21T21:59:16.295383+0000","last_in_change":"2024-08-21T21:59:09.468854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-21T21:59:16.562763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-21T21:59:16.991606+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d9652f75-1900-4c11-9db5-f3adedac6cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20064},{"type":"v1","addr":"172.21.15.178:6817","nonce":20064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20064},{"type":"v1","addr":"172.21.15.178:6819","nonce":20064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20064},{"type":"v1","addr":"172.21.15.178:6823","nonce":20064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20064},{"type":"v1","addr":"172.21.15.178:6821","nonce":20064}]},"public_addr":"172.21.15.178:6817/20064","cluster_addr":"172.21.15.178:6819/20064","heartbeat_back_addr":"172.21.15.178:6823/20064","heartbeat_front_addr":"172.21.15.178:6821/20064","state":["exists","up"]},{"osd":1,"uuid":"929d5722-725a-4d2c-a9fc-563930bf59d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20054},{"type":"v1","addr":"172.21.15.178:6801","nonce":20054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20054},{"type":"v1","addr":"172.21.15.178:6803","nonce":20054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":20054},{"type":"v1","addr":"172.21.15.178:6807","nonce":20054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20054},{"type":"v1","addr":"172.21.15.178:6805","nonce":20054}]},"public_addr":"172.21.15.178:6801/20054","cluster_addr":"172.21.15.178:6803/20054","heartbeat_back_addr":"172.21.15.178:6807/20054","heartbeat_front_addr":"172.21.15.178:6805/20054","state":["exists","up"]},{"osd":2,"uuid":"14fd26b5-9422-4d68-bb43-ce789449b6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20055},{"type":"v1","addr":"172.21.15.178:6809","nonce":20055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":20055},{"type":"v1","addr":"172.21.15.178:6811","nonce":20055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20055},{"type":"v1","addr":"172.21.15.178:6815","nonce":20055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20055},{"type":"v1","addr":"172.21.15.178:6813","nonce":20055}]},"public_addr":"172.21.15.178:6809/20055","cluster_addr":"172.21.15.178:6811/20055","heartbeat_back_addr":"172.21.15.178:6815/20055","heartbeat_front_addr":"172.21.15.178:6813/20055","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:59:13.863814+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:59:13.787173+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:59:13.846804+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:59:20.965 INFO:tasks.ceph.ceph_manager.c2:all up! 2024-08-21T21:59:20.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-08-21T21:59:21.224 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:59:21.225 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":12,"fsid":"963999ae-000a-4464-a440-8bde81eb6b50","created":"2024-08-21T21:59:08.600909+0000","modified":"2024-08-21T21:59:20.300542+0000","last_up_change":"2024-08-21T21:59:16.295383+0000","last_in_change":"2024-08-21T21:59:09.468854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-21T21:59:16.562763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-21T21:59:16.991606+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d9652f75-1900-4c11-9db5-f3adedac6cf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":20064},{"type":"v1","addr":"172.21.15.178:6817","nonce":20064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":20064},{"type":"v1","addr":"172.21.15.178:6819","nonce":20064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":20064},{"type":"v1","addr":"172.21.15.178:6823","nonce":20064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":20064},{"type":"v1","addr":"172.21.15.178:6821","nonce":20064}]},"public_addr":"172.21.15.178:6817/20064","cluster_addr":"172.21.15.178:6819/20064","heartbeat_back_addr":"172.21.15.178:6823/20064","heartbeat_front_addr":"172.21.15.178:6821/20064","state":["exists","up"]},{"osd":1,"uuid":"929d5722-725a-4d2c-a9fc-563930bf59d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":20054},{"type":"v1","addr":"172.21.15.178:6801","nonce":20054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":20054},{"type":"v1","addr":"172.21.15.178:6803","nonce":20054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":20054},{"type":"v1","addr":"172.21.15.178:6807","nonce":20054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":20054},{"type":"v1","addr":"172.21.15.178:6805","nonce":20054}]},"public_addr":"172.21.15.178:6801/20054","cluster_addr":"172.21.15.178:6803/20054","heartbeat_back_addr":"172.21.15.178:6807/20054","heartbeat_front_addr":"172.21.15.178:6805/20054","state":["exists","up"]},{"osd":2,"uuid":"14fd26b5-9422-4d68-bb43-ce789449b6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":20055},{"type":"v1","addr":"172.21.15.178:6809","nonce":20055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":20055},{"type":"v1","addr":"172.21.15.178:6811","nonce":20055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":20055},{"type":"v1","addr":"172.21.15.178:6815","nonce":20055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":20055},{"type":"v1","addr":"172.21.15.178:6813","nonce":20055}]},"public_addr":"172.21.15.178:6809/20055","cluster_addr":"172.21.15.178:6811/20055","heartbeat_back_addr":"172.21.15.178:6815/20055","heartbeat_front_addr":"172.21.15.178:6813/20055","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:59:13.863814+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:59:13.787173+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-21T21:59:13.846804+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-21T21:59:21.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2024-08-21T21:59:21.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2024-08-21T21:59:21.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2024-08-21T21:59:21.367 INFO:teuthology.orchestra.run.smithi178.stdout:34359738371 2024-08-21T21:59:21.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-08-21T21:59:21.369 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-08-21T21:59:21.369 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-08-21T21:59:21.370 INFO:teuthology.orchestra.run.smithi178.stdout:34359738371 2024-08-21T21:59:21.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-08-21T21:59:21.682 INFO:teuthology.orchestra.run.smithi178.stdout:34359738371 2024-08-21T21:59:21.687 INFO:teuthology.orchestra.run.smithi178.stdout:34359738370 2024-08-21T21:59:21.690 INFO:teuthology.orchestra.run.smithi178.stdout:34359738370 2024-08-21T21:59:21.695 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.2 2024-08-21T21:59:21.701 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2024-08-21T21:59:21.703 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2024-08-21T21:59:22.696 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-08-21T21:59:22.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-08-21T21:59:22.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-08-21T21:59:22.985 INFO:teuthology.orchestra.run.smithi178.stdout:34359738371 2024-08-21T21:59:22.998 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.2 2024-08-21T21:59:23.019 INFO:teuthology.orchestra.run.smithi178.stdout:34359738370 2024-08-21T21:59:23.024 INFO:teuthology.orchestra.run.smithi178.stdout:34359738370 2024-08-21T21:59:23.032 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2024-08-21T21:59:23.036 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.1 2024-08-21T21:59:23.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2024-08-21T21:59:24.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2024-08-21T21:59:24.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2024-08-21T21:59:24.293 INFO:teuthology.orchestra.run.smithi178.stdout:34359738373 2024-08-21T21:59:24.305 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738373 for osd.2 2024-08-21T21:59:24.306 DEBUG:teuthology.parallel:result is None 2024-08-21T21:59:24.320 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-08-21T21:59:24.332 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.0 2024-08-21T21:59:24.333 INFO:teuthology.orchestra.run.smithi178.stdout:34359738372 2024-08-21T21:59:24.333 DEBUG:teuthology.parallel:result is None 2024-08-21T21:59:24.344 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738372 for osd.1 2024-08-21T21:59:24.345 DEBUG:teuthology.parallel:result is None 2024-08-21T21:59:24.345 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2024-08-21T21:59:24.345 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-08-21T21:59:24.645 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:59:24.646 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-08-21T21:59:24.657 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-08-21T21:59:24.535373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893812,"kb_used_data":1476,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":313678988,"statfs":{"total":322122547200,"available":321207283712,"internally_reserved":0,"allocated":1511424,"data_stored":1269197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000322"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.308899+0000","last_change":"2024-08-21T21:59:20.309078+0000","last_active":"2024-08-21T21:59:20.308899+0000","last_peered":"2024-08-21T21:59:20.308899+0000","last_clean":"2024-08-21T21:59:20.308899+0000","last_became_active":"2024-08-21T21:59:18.303052+0000","last_became_peered":"2024-08-21T21:59:18.303052+0000","last_unstale":"2024-08-21T21:59:20.308899+0000","last_undegraded":"2024-08-21T21:59:20.308899+0000","last_fullsized":"2024-08-21T21:59:20.308899+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T07:28:36.498195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043046099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.309112+0000","last_change":"2024-08-21T21:59:20.309502+0000","last_active":"2024-08-21T21:59:20.309112+0000","last_peered":"2024-08-21T21:59:20.309112+0000","last_clean":"2024-08-21T21:59:20.309112+0000","last_became_active":"2024-08-21T21:59:18.304778+0000","last_became_peered":"2024-08-21T21:59:18.304778+0000","last_unstale":"2024-08-21T21:59:20.309112+0000","last_undegraded":"2024-08-21T21:59:20.309112+0000","last_fullsized":"2024-08-21T21:59:20.309112+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T06:20:23.011484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086921999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.309151+0000","last_change":"2024-08-21T21:59:20.309349+0000","last_active":"2024-08-21T21:59:20.309151+0000","last_peered":"2024-08-21T21:59:20.309151+0000","last_clean":"2024-08-21T21:59:20.309151+0000","last_became_active":"2024-08-21T21:59:18.304287+0000","last_became_peered":"2024-08-21T21:59:18.304287+0000","last_unstale":"2024-08-21T21:59:20.309151+0000","last_undegraded":"2024-08-21T21:59:20.309151+0000","last_fullsized":"2024-08-21T21:59:20.309151+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T23:55:16.199268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065879700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.308850+0000","last_change":"2024-08-21T21:59:20.309068+0000","last_active":"2024-08-21T21:59:20.308850+0000","last_peered":"2024-08-21T21:59:20.308850+0000","last_clean":"2024-08-21T21:59:20.308850+0000","last_became_active":"2024-08-21T21:59:18.304253+0000","last_became_peered":"2024-08-21T21:59:18.304253+0000","last_unstale":"2024-08-21T21:59:20.308850+0000","last_undegraded":"2024-08-21T21:59:20.308850+0000","last_fullsized":"2024-08-21T21:59:20.308850+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T05:17:34.409574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000459803,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"12'2","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.307586+0000","last_change":"2024-08-21T21:59:20.307586+0000","last_active":"2024-08-21T21:59:20.307586+0000","last_peered":"2024-08-21T21:59:20.307586+0000","last_clean":"2024-08-21T21:59:20.307586+0000","last_became_active":"2024-08-21T21:59:18.303038+0000","last_became_peered":"2024-08-21T21:59:18.303038+0000","last_unstale":"2024-08-21T21:59:20.307586+0000","last_undegraded":"2024-08-21T21:59:20.307586+0000","last_fullsized":"2024-08-21T21:59:20.307586+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T03:11:41.563131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092943799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.303211+0000","last_change":"2024-08-21T21:59:20.303483+0000","last_active":"2024-08-21T21:59:20.303211+0000","last_peered":"2024-08-21T21:59:20.303211+0000","last_clean":"2024-08-21T21:59:20.303211+0000","last_became_active":"2024-08-21T21:59:18.303547+0000","last_became_peered":"2024-08-21T21:59:18.303547+0000","last_unstale":"2024-08-21T21:59:20.303211+0000","last_undegraded":"2024-08-21T21:59:20.303211+0000","last_fullsized":"2024-08-21T21:59:20.303211+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T03:11:41.563131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041461599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.303264+0000","last_change":"2024-08-21T21:59:20.303490+0000","last_active":"2024-08-21T21:59:20.303264+0000","last_peered":"2024-08-21T21:59:20.303264+0000","last_clean":"2024-08-21T21:59:20.303264+0000","last_became_active":"2024-08-21T21:59:18.305337+0000","last_became_peered":"2024-08-21T21:59:18.305337+0000","last_unstale":"2024-08-21T21:59:20.303264+0000","last_undegraded":"2024-08-21T21:59:20.303264+0000","last_fullsized":"2024-08-21T21:59:20.303264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T07:27:38.129229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040025899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"10'1","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.309481+0000","last_change":"2024-08-21T21:59:20.309706+0000","last_active":"2024-08-21T21:59:20.309481+0000","last_peered":"2024-08-21T21:59:20.309481+0000","last_clean":"2024-08-21T21:59:20.309481+0000","last_became_active":"2024-08-21T21:59:18.303736+0000","last_became_peered":"2024-08-21T21:59:18.303736+0000","last_unstale":"2024-08-21T21:59:20.309481+0000","last_undegraded":"2024-08-21T21:59:20.309481+0000","last_fullsized":"2024-08-21T21:59:20.309481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T08:16:41.445473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086388800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-08-21T21:59:19.301828+0000","last_change":"2024-08-21T21:59:18.303313+0000","last_active":"2024-08-21T21:59:19.301828+0000","last_peered":"2024-08-21T21:59:19.301828+0000","last_clean":"2024-08-21T21:59:19.301828+0000","last_became_active":"2024-08-21T21:59:18.303067+0000","last_became_peered":"2024-08-21T21:59:18.303067+0000","last_unstale":"2024-08-21T21:59:19.301828+0000","last_undegraded":"2024-08-21T21:59:19.301828+0000","last_fullsized":"2024-08-21T21:59:19.301828+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T08:05:28.785271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297460,"kb_used_data":100,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104560140,"statfs":{"total":107374182400,"available":107069583360,"internally_reserved":0,"allocated":102400,"data_stored":28739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298176,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559424,"statfs":{"total":107374182400,"available":107068850176,"internally_reserved":0,"allocated":704512,"data_stored":620229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.186}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298176,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559424,"statfs":{"total":107374182400,"available":107068850176,"internally_reserved":0,"allocated":704512,"data_stored":620229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-21T21:59:24.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2024-08-21T21:59:24.913 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:59:24.914 INFO:teuthology.orchestra.run.smithi178.stderr:dumped all 2024-08-21T21:59:24.925 INFO:teuthology.orchestra.run.smithi178.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-08-21T21:59:24.535373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":893812,"kb_used_data":1476,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":313678988,"statfs":{"total":322122547200,"available":321207283712,"internally_reserved":0,"allocated":1511424,"data_stored":1269197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000322"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.308899+0000","last_change":"2024-08-21T21:59:20.309078+0000","last_active":"2024-08-21T21:59:20.308899+0000","last_peered":"2024-08-21T21:59:20.308899+0000","last_clean":"2024-08-21T21:59:20.308899+0000","last_became_active":"2024-08-21T21:59:18.303052+0000","last_became_peered":"2024-08-21T21:59:18.303052+0000","last_unstale":"2024-08-21T21:59:20.308899+0000","last_undegraded":"2024-08-21T21:59:20.308899+0000","last_fullsized":"2024-08-21T21:59:20.308899+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T07:28:36.498195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043046099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.309112+0000","last_change":"2024-08-21T21:59:20.309502+0000","last_active":"2024-08-21T21:59:20.309112+0000","last_peered":"2024-08-21T21:59:20.309112+0000","last_clean":"2024-08-21T21:59:20.309112+0000","last_became_active":"2024-08-21T21:59:18.304778+0000","last_became_peered":"2024-08-21T21:59:18.304778+0000","last_unstale":"2024-08-21T21:59:20.309112+0000","last_undegraded":"2024-08-21T21:59:20.309112+0000","last_fullsized":"2024-08-21T21:59:20.309112+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T06:20:23.011484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086921999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.309151+0000","last_change":"2024-08-21T21:59:20.309349+0000","last_active":"2024-08-21T21:59:20.309151+0000","last_peered":"2024-08-21T21:59:20.309151+0000","last_clean":"2024-08-21T21:59:20.309151+0000","last_became_active":"2024-08-21T21:59:18.304287+0000","last_became_peered":"2024-08-21T21:59:18.304287+0000","last_unstale":"2024-08-21T21:59:20.309151+0000","last_undegraded":"2024-08-21T21:59:20.309151+0000","last_fullsized":"2024-08-21T21:59:20.309151+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-22T23:55:16.199268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065879700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.308850+0000","last_change":"2024-08-21T21:59:20.309068+0000","last_active":"2024-08-21T21:59:20.308850+0000","last_peered":"2024-08-21T21:59:20.308850+0000","last_clean":"2024-08-21T21:59:20.308850+0000","last_became_active":"2024-08-21T21:59:18.304253+0000","last_became_peered":"2024-08-21T21:59:18.304253+0000","last_unstale":"2024-08-21T21:59:20.308850+0000","last_undegraded":"2024-08-21T21:59:20.308850+0000","last_fullsized":"2024-08-21T21:59:20.308850+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T05:17:34.409574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000459803,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"12'2","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.307586+0000","last_change":"2024-08-21T21:59:20.307586+0000","last_active":"2024-08-21T21:59:20.307586+0000","last_peered":"2024-08-21T21:59:20.307586+0000","last_clean":"2024-08-21T21:59:20.307586+0000","last_became_active":"2024-08-21T21:59:18.303038+0000","last_became_peered":"2024-08-21T21:59:18.303038+0000","last_unstale":"2024-08-21T21:59:20.307586+0000","last_undegraded":"2024-08-21T21:59:20.307586+0000","last_fullsized":"2024-08-21T21:59:20.307586+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T03:11:41.563131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092943799999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.303211+0000","last_change":"2024-08-21T21:59:20.303483+0000","last_active":"2024-08-21T21:59:20.303211+0000","last_peered":"2024-08-21T21:59:20.303211+0000","last_clean":"2024-08-21T21:59:20.303211+0000","last_became_active":"2024-08-21T21:59:18.303547+0000","last_became_peered":"2024-08-21T21:59:18.303547+0000","last_unstale":"2024-08-21T21:59:20.303211+0000","last_undegraded":"2024-08-21T21:59:20.303211+0000","last_fullsized":"2024-08-21T21:59:20.303211+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T03:11:41.563131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041461599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.303264+0000","last_change":"2024-08-21T21:59:20.303490+0000","last_active":"2024-08-21T21:59:20.303264+0000","last_peered":"2024-08-21T21:59:20.303264+0000","last_clean":"2024-08-21T21:59:20.303264+0000","last_became_active":"2024-08-21T21:59:18.305337+0000","last_became_peered":"2024-08-21T21:59:18.305337+0000","last_unstale":"2024-08-21T21:59:20.303264+0000","last_undegraded":"2024-08-21T21:59:20.303264+0000","last_fullsized":"2024-08-21T21:59:20.303264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T07:27:38.129229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040025899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"10'1","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-08-21T21:59:20.309481+0000","last_change":"2024-08-21T21:59:20.309706+0000","last_active":"2024-08-21T21:59:20.309481+0000","last_peered":"2024-08-21T21:59:20.309481+0000","last_clean":"2024-08-21T21:59:20.309481+0000","last_became_active":"2024-08-21T21:59:18.303736+0000","last_became_peered":"2024-08-21T21:59:18.303736+0000","last_unstale":"2024-08-21T21:59:20.309481+0000","last_undegraded":"2024-08-21T21:59:20.309481+0000","last_fullsized":"2024-08-21T21:59:20.309481+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T08:16:41.445473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086388800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-08-21T21:59:19.301828+0000","last_change":"2024-08-21T21:59:18.303313+0000","last_active":"2024-08-21T21:59:19.301828+0000","last_peered":"2024-08-21T21:59:19.301828+0000","last_clean":"2024-08-21T21:59:19.301828+0000","last_became_active":"2024-08-21T21:59:18.303067+0000","last_became_peered":"2024-08-21T21:59:18.303067+0000","last_unstale":"2024-08-21T21:59:19.301828+0000","last_undegraded":"2024-08-21T21:59:19.301828+0000","last_fullsized":"2024-08-21T21:59:19.301828+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-21T21:59:17.296214+0000","last_clean_scrub_stamp":"2024-08-21T21:59:17.296214+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-23T08:05:28.785271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297460,"kb_used_data":100,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104560140,"statfs":{"total":107374182400,"available":107069583360,"internally_reserved":0,"allocated":102400,"data_stored":28739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298176,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559424,"statfs":{"total":107374182400,"available":107068850176,"internally_reserved":0,"allocated":704512,"data_stored":620229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.186}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":298176,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":104559424,"statfs":{"total":107374182400,"available":107068850176,"internally_reserved":0,"allocated":704512,"data_stored":620229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-21T21:59:24.926 INFO:tasks.ceph.ceph_manager.c2:clean! 2024-08-21T21:59:24.926 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2024-08-21T21:59:24.926 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2024-08-21T21:59:24.927 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2024-08-21T21:59:25.214 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-08-21T21:59:25.214 INFO:teuthology.orchestra.run.smithi178.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-21T21:59:25.226 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2024-08-21T21:59:25.226 INFO:teuthology.run_tasks:Running task rgw... 2024-08-21T21:59:25.241 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2024-08-21T21:59:25.241 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2024-08-21T21:59:25.242 INFO:tasks.rgw:Creating data pools 2024-08-21T21:59:25.242 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2024-08-21T21:59:25.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-08-21T21:59:26.437 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'default.rgw.buckets.data' created 2024-08-21T21:59:26.459 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-08-21T21:59:27.440 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-21T21:59:27.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-08-21T21:59:28.455 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'default.rgw.buckets.index' created 2024-08-21T21:59:28.476 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-08-21T21:59:29.452 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-21T21:59:29.484 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2024-08-21T21:59:29.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2024-08-21T21:59:29.746 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'default.rgw.buckets.data' already exists 2024-08-21T21:59:29.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2024-08-21T21:59:30.457 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-21T21:59:30.470 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2024-08-21T21:59:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'default.rgw.buckets.index' already exists 2024-08-21T21:59:30.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2024-08-21T21:59:31.458 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-21T21:59:31.471 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2024-08-21T21:59:31.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-08-21T21:59:32.612 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'default.rgw.buckets.data' created 2024-08-21T21:59:32.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-08-21T21:59:33.616 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-21T21:59:33.643 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-08-21T21:59:34.618 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'default.rgw.buckets.index' created 2024-08-21T21:59:34.639 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-08-21T21:59:35.621 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-21T21:59:35.654 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2024-08-21T21:59:35.654 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2024-08-21T21:59:35.919 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'default.rgw.buckets.data' already exists 2024-08-21T21:59:35.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2024-08-21T21:59:36.625 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-08-21T21:59:36.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2024-08-21T21:59:36.902 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'default.rgw.buckets.index' already exists 2024-08-21T21:59:36.912 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2024-08-21T21:59:37.626 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-08-21T21:59:37.641 DEBUG:tasks.rgw:Pools created 2024-08-21T21:59:37.641 INFO:tasks.rgw:Configuring compression type = random 2024-08-21T21:59:37.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2024-08-21T21:59:37.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2024-08-21T21:59:37.642 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2024-08-21T21:59:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:37.715 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:37.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:37.723+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.488+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.488+0000 7f0054e087c0 20 realm 2024-08-21T21:59:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.488+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-21T21:59:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 10 Creating default zonegroup 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.489+0000 7f0054e087c0 20 get_system_obj_state: rctx=0x7ffcea2208a0 obj=:.pools.avail state=0x55ca4acfbf20 s->prefetch_data=0 2024-08-21T21:59:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.490+0000 7f0054e087c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-08-21T21:59:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.498+0000 7f0054e087c0 10 WARNING: store->list_zones() returned r=-2 2024-08-21T21:59:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.498+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.498+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.502+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.503+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.503+0000 7f0054e087c0 10 could not read realm id: (2) No such file or directory 2024-08-21T21:59:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.503+0000 7f0054e087c0 10 WARNING: failed to set zone as default, r=-22 2024-08-21T21:59:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.503+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.503+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.508+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.508+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.508+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.509+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.509+0000 7f0054e087c0 10 Using default name default 2024-08-21T21:59:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.509+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.510+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.517+0000 7f0054e087c0 20 RGWRados::pool_iterate: got zone_info.a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b 2024-08-21T21:59:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.517+0000 7f0054e087c0 20 RGWRados::pool_iterate: got zonegroup_info.0e1b858e-f944-4883-8951-3d31fce7636f 2024-08-21T21:59:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.517+0000 7f0054e087c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.517+0000 7f0054e087c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.517+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 20 zone default found 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.518+0000 7f0054e087c0 4 Realm: () 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.519+0000 7f0054e087c0 4 ZoneGroup: default (0e1b858e-f944-4883-8951-3d31fce7636f) 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.519+0000 7f0054e087c0 4 Zone: default (a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b) 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.519+0000 7f0054e087c0 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.519+0000 7f0054e087c0 20 zonegroup default 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.519+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.519+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:39.519+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:41.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:41.492+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:41.493+0000 7f0054e087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:41.493+0000 7f0054e087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:41.493+0000 7f0054e087c0 20 started sync module instance, tier type = 2024-08-21T21:59:41.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:41.493+0000 7f0054e087c0 20 started zone id=a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b (name=default) with tier type = 2024-08-21T21:59:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.532+0000 7f0054e087c0 20 add_watcher() i=0 2024-08-21T21:59:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.532+0000 7f0054e087c0 20 add_watcher() i=1 2024-08-21T21:59:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.532+0000 7f0054e087c0 20 add_watcher() i=2 2024-08-21T21:59:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.532+0000 7f0054e087c0 20 add_watcher() i=3 2024-08-21T21:59:43.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.532+0000 7f0054e087c0 20 add_watcher() i=4 2024-08-21T21:59:43.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.532+0000 7f0054e087c0 20 add_watcher() i=5 2024-08-21T21:59:43.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.532+0000 7f0054e087c0 20 add_watcher() i=6 2024-08-21T21:59:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.533+0000 7f0054e087c0 20 add_watcher() i=7 2024-08-21T21:59:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.533+0000 7f0054e087c0 2 all 8 watchers are set, enabling cache 2024-08-21T21:59:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-08-21T21:59:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.534+0000 7f0054e087c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.535+0000 7f0054e087c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-08-21T21:59:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.535+0000 7f0054e087c0 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1205 entering 2024-08-21T21:59:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.536+0000 7f0054e087c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.539+0000 7eff9ffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.539+0000 7f0054e087c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:43.539+0000 7f0054e087c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:46.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.505+0000 7f0054e087c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-08-21T21:59:46.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.505+0000 7f0054e087c0 5 note: GC not initialized 2024-08-21T21:59:46.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.506+0000 7eff8affd640 20 reqs_thread_entry: start 2024-08-21T21:59:46.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.506+0000 7f0054e087c0 20 init_complete bucket index max shards: 11 2024-08-21T21:59:46.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.506+0000 7f0054e087c0 10 rgw notify: Started notification manager with: 1 workers 2024-08-21T21:59:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.506+0000 7eff88ff9640 20 rgw notify: INFO: next queues processing will happen at: Wed Aug 21 22:00:16 2024 2024-08-21T21:59:46.508 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T21:59:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.506+0000 7eff7bfff640 20 reqs_thread_entry: start 2024-08-21T21:59:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.529+0000 7f0054e087c0 20 remove_watcher() i=0 2024-08-21T21:59:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.529+0000 7f0054e087c0 2 removed watcher, disabling cache 2024-08-21T21:59:46.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.530+0000 7f0054e087c0 20 remove_watcher() i=1 2024-08-21T21:59:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.531+0000 7f0054e087c0 20 remove_watcher() i=2 2024-08-21T21:59:46.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.534+0000 7f0054e087c0 20 remove_watcher() i=3 2024-08-21T21:59:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.535+0000 7f0054e087c0 20 remove_watcher() i=4 2024-08-21T21:59:46.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.537+0000 7f0054e087c0 20 remove_watcher() i=5 2024-08-21T21:59:46.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.539+0000 7f0054e087c0 20 remove_watcher() i=6 2024-08-21T21:59:46.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.540+0000 7f0054e087c0 20 remove_watcher() i=7 2024-08-21T21:59:46.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.540+0000 7eff9ffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:46.545 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-08-21T21:59:46.556 DEBUG:tasks.util.rgw: json result: [] 2024-08-21T21:59:46.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:46.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:46.557 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-21T21:59:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.612+0000 7f2ec03af7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.612+0000 7f2ec03af7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.615+0000 7f2e077fe640 20 reqs_thread_entry: start 2024-08-21T21:59:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.615+0000 7f2ec03af7c0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-08-21T21:59:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.615+0000 7f2ec03af7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.616+0000 7f2ec03af7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.616+0000 7f2ec03af7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.617+0000 7f2ec03af7c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.623+0000 7f2ec03af7c0 20 RGWRados::pool_iterate: got zone_info.a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.623+0000 7f2ec03af7c0 20 RGWRados::pool_iterate: got zonegroup_info.0e1b858e-f944-4883-8951-3d31fce7636f 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.623+0000 7f2ec03af7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.623+0000 7f2ec03af7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.623+0000 7f2ec03af7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.624+0000 7f2ec03af7c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.624+0000 7f2ec03af7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.624+0000 7f2ec03af7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.624+0000 7f2ec03af7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.624+0000 7f2ec03af7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.624+0000 7f2ec03af7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.624+0000 7f2ec03af7c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b", 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default", 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "default.rgw.meta:root", 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "default.rgw.control", 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "default.rgw.log", 2024-08-21T21:59:46.630 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "default.rgw.otp", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "", 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "" 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T21:59:46.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "compression_type": "random" 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T21:59:46.632 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "", 2024-08-21T21:59:46.633 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-21T21:59:46.633 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T21:59:46.633 DEBUG:tasks.util.rgw: json result: {'id': 'a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-21T21:59:46.634 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2024-08-21T21:59:46.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2024-08-21T21:59:46.634 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2024-08-21T21:59:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:46.673 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:46.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.679+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.680+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.680+0000 7f0e993207c0 20 realm 2024-08-21T21:59:46.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.680+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.680+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.680+0000 7f0e993207c0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-21T21:59:46.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.680+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.680+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.681+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.681+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.681+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.682+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:46.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.682+0000 7f0e993207c0 20 searching for the correct realm 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.688+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zone_info.a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.688+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zonegroup_info.0e1b858e-f944-4883-8951-3d31fce7636f 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.688+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.688+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.688+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.689+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.690+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.695+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zone_info.a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b 2024-08-21T21:59:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.695+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zonegroup_info.0e1b858e-f944-4883-8951-3d31fce7636f 2024-08-21T21:59:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.695+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.695+0000 7f0e993207c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.695+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.695+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.695+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 zone default found 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 4 Realm: () 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 4 ZoneGroup: default (0e1b858e-f944-4883-8951-3d31fce7636f) 2024-08-21T21:59:46.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 4 Zone: default (a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b) 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 zonegroup default 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.696+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.697+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.697+0000 7f0e993207c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.697+0000 7f0e993207c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.697+0000 7f0e993207c0 20 started sync module instance, tier type = 2024-08-21T21:59:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.697+0000 7f0e993207c0 20 started zone id=a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b (name=default) with tier type = 2024-08-21T21:59:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.715+0000 7f0e993207c0 20 add_watcher() i=0 2024-08-21T21:59:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.715+0000 7f0e993207c0 20 add_watcher() i=1 2024-08-21T21:59:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.715+0000 7f0e993207c0 20 add_watcher() i=2 2024-08-21T21:59:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.715+0000 7f0e993207c0 20 add_watcher() i=3 2024-08-21T21:59:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.715+0000 7f0e993207c0 20 add_watcher() i=4 2024-08-21T21:59:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.715+0000 7f0e993207c0 20 add_watcher() i=5 2024-08-21T21:59:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.715+0000 7f0e993207c0 20 add_watcher() i=6 2024-08-21T21:59:46.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.718+0000 7f0e993207c0 20 add_watcher() i=7 2024-08-21T21:59:46.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.718+0000 7f0e993207c0 2 all 8 watchers are set, enabling cache 2024-08-21T21:59:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.719+0000 7f0e667e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.719+0000 7f0e993207c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.719+0000 7f0e993207c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.723+0000 7f0e993207c0 5 note: GC not initialized 2024-08-21T21:59:46.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.723+0000 7f0dd57fa640 20 reqs_thread_entry: start 2024-08-21T21:59:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.723+0000 7f0dceffd640 20 reqs_thread_entry: start 2024-08-21T21:59:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.723+0000 7f0e993207c0 20 init_complete bucket index max shards: 11 2024-08-21T21:59:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.723+0000 7f0e993207c0 10 rgw notify: Started notification manager with: 1 workers 2024-08-21T21:59:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.723+0000 7f0dcd7fa640 20 rgw notify: INFO: next queues processing will happen at: Wed Aug 21 22:00:17 2024 2024-08-21T21:59:46.725 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T21:59:46.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.723+0000 7f0dccff9640 20 reqs_thread_entry: start 2024-08-21T21:59:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.731+0000 7f0e993207c0 20 remove_watcher() i=0 2024-08-21T21:59:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.731+0000 7f0e993207c0 2 removed watcher, disabling cache 2024-08-21T21:59:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.732+0000 7f0e993207c0 20 remove_watcher() i=1 2024-08-21T21:59:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.733+0000 7f0e993207c0 20 remove_watcher() i=2 2024-08-21T21:59:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.733+0000 7f0e993207c0 20 remove_watcher() i=3 2024-08-21T21:59:46.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.734+0000 7f0e993207c0 20 remove_watcher() i=4 2024-08-21T21:59:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.735+0000 7f0e993207c0 20 remove_watcher() i=5 2024-08-21T21:59:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.736+0000 7f0e993207c0 20 remove_watcher() i=6 2024-08-21T21:59:46.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.737+0000 7f0e993207c0 20 remove_watcher() i=7 2024-08-21T21:59:46.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.737+0000 7f0e667e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:46.741 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-08-21T21:59:46.743 DEBUG:tasks.util.rgw: json result: [] 2024-08-21T21:59:46.743 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:46.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:46.743 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-21T21:59:46.830 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:46.830 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:46.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.836+0000 7faea93447c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:46.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.836+0000 7faea93447c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:46.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.838+0000 7fadeffff640 20 reqs_thread_entry: start 2024-08-21T21:59:46.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.838+0000 7faea93447c0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-08-21T21:59:46.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.839+0000 7faea93447c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.839+0000 7faea93447c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.839+0000 7faea93447c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.840+0000 7faea93447c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.846+0000 7faea93447c0 20 RGWRados::pool_iterate: got zone_info.a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b 2024-08-21T21:59:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.846+0000 7faea93447c0 20 RGWRados::pool_iterate: got zonegroup_info.0e1b858e-f944-4883-8951-3d31fce7636f 2024-08-21T21:59:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.846+0000 7faea93447c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.846+0000 7faea93447c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.846+0000 7faea93447c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.846+0000 7faea93447c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:46.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.846+0000 7faea93447c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.847+0000 7faea93447c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.847+0000 7faea93447c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.847+0000 7faea93447c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.847+0000 7faea93447c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:46.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:46.847+0000 7faea93447c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:46.852 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T21:59:46.852 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b", 2024-08-21T21:59:46.852 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default", 2024-08-21T21:59:46.852 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "default.rgw.meta:root", 2024-08-21T21:59:46.852 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "default.rgw.control", 2024-08-21T21:59:46.852 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "default.rgw.log", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "default.rgw.otp", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "", 2024-08-21T21:59:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "" 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "compression_type": "random" 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T21:59:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T21:59:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T21:59:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "", 2024-08-21T21:59:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-21T21:59:46.855 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T21:59:46.856 DEBUG:tasks.util.rgw: json result: {'id': 'a88e3c4f-5c5a-4e81-94cf-f61ae2be8b1b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-21T21:59:46.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2024-08-21T21:59:46.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2024-08-21T21:59:46.856 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2024-08-21T21:59:46.927 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:46.927 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:46.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:46.935+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.648+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.648+0000 7f9ddb5337c0 20 realm 2024-08-21T21:59:48.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.649+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.649+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.649+0000 7f9ddb5337c0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-21T21:59:48.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.649+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.650+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.650+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.657+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.657+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.658+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.658+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.658+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.658+0000 7f9ddb5337c0 10 Creating default zonegroup 2024-08-21T21:59:48.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.658+0000 7f9ddb5337c0 20 get_system_obj_state: rctx=0x7ffc7fdc1df0 obj=:.pools.avail state=0x55ba39313f20 s->prefetch_data=0 2024-08-21T21:59:48.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.659+0000 7f9ddb5337c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2024-08-21T21:59:48.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.668+0000 7f9ddb5337c0 10 WARNING: store->list_zones() returned r=-2 2024-08-21T21:59:48.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.668+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.668+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.671+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.673 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.671+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.671+0000 7f9ddb5337c0 10 could not read realm id: (2) No such file or directory 2024-08-21T21:59:48.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.671+0000 7f9ddb5337c0 10 WARNING: failed to set zone as default, r=-22 2024-08-21T21:59:48.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.671+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.671+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.676 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.673+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.676 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.674+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:48.676 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.674+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.674+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:48.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.674+0000 7f9ddb5337c0 10 Using default name default 2024-08-21T21:59:48.677 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.674+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.674+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.674+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.675+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.675+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.675+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.675+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.675+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.675+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.675+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:48.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.682+0000 7f9ddb5337c0 20 RGWRados::pool_iterate: got zone_info.b607c794-1dc4-42ec-9ba3-bfb1b2f779e7 2024-08-21T21:59:48.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.682+0000 7f9ddb5337c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:48.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.682+0000 7f9ddb5337c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:48.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.682+0000 7f9ddb5337c0 20 RGWRados::pool_iterate: got zonegroup_info.664b2903-5133-4ce7-b8a9-22c479b12fdd 2024-08-21T21:59:48.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.682+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.683+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:48.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.683+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.683+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.683+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.683+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.683+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.683+0000 7f9ddb5337c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 20 zone default found 2024-08-21T21:59:48.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 4 Realm: () 2024-08-21T21:59:48.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 4 ZoneGroup: default (664b2903-5133-4ce7-b8a9-22c479b12fdd) 2024-08-21T21:59:48.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 4 Zone: default (b607c794-1dc4-42ec-9ba3-bfb1b2f779e7) 2024-08-21T21:59:48.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-21T21:59:48.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 20 zonegroup default 2024-08-21T21:59:48.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:48.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:48.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:48.684+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:50.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:50.662+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:50.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:50.662+0000 7f9ddb5337c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:50.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:50.663+0000 7f9ddb5337c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:50.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:50.663+0000 7f9ddb5337c0 20 started sync module instance, tier type = 2024-08-21T21:59:50.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:50.663+0000 7f9ddb5337c0 20 started zone id=b607c794-1dc4-42ec-9ba3-bfb1b2f779e7 (name=default) with tier type = 2024-08-21T21:59:52.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.685+0000 7f9ddb5337c0 20 add_watcher() i=0 2024-08-21T21:59:52.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.685+0000 7f9ddb5337c0 20 add_watcher() i=1 2024-08-21T21:59:52.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.685+0000 7f9ddb5337c0 20 add_watcher() i=2 2024-08-21T21:59:52.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.685+0000 7f9ddb5337c0 20 add_watcher() i=3 2024-08-21T21:59:52.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.685+0000 7f9ddb5337c0 20 add_watcher() i=4 2024-08-21T21:59:52.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.685+0000 7f9ddb5337c0 20 add_watcher() i=5 2024-08-21T21:59:52.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.685+0000 7f9ddb5337c0 20 add_watcher() i=6 2024-08-21T21:59:52.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 add_watcher() i=7 2024-08-21T21:59:52.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 2 all 8 watchers are set, enabling cache 2024-08-21T21:59:52.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2024-08-21T21:59:52.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2024-08-21T21:59:52.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:52.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.686+0000 7f9ddb5337c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.687+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.687+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.687+0000 7f9ddb5337c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.687+0000 7f9ddb5337c0 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2024-08-21T21:59:52.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.687+0000 7f9ddb5337c0 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1205 entering 2024-08-21T21:59:52.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.688+0000 7f9ddb5337c0 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1163 entering 2024-08-21T21:59:52.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.690+0000 7f9daa7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:52.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.690+0000 7f9ddb5337c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:52.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:52.690+0000 7f9ddb5337c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:55.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.664+0000 7f9ddb5337c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2024-08-21T21:59:55.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.664+0000 7f9ddb5337c0 5 note: GC not initialized 2024-08-21T21:59:55.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.665+0000 7f9d157fa640 20 reqs_thread_entry: start 2024-08-21T21:59:55.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.665+0000 7f9ddb5337c0 20 init_complete bucket index max shards: 11 2024-08-21T21:59:55.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.665+0000 7f9ddb5337c0 10 rgw notify: Started notification manager with: 1 workers 2024-08-21T21:59:55.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.665+0000 7f9d06ffd640 20 rgw notify: INFO: next queues processing will happen at: Wed Aug 21 22:00:25 2024 2024-08-21T21:59:55.667 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-08-21T21:59:55.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.665+0000 7f9d067fc640 20 reqs_thread_entry: start 2024-08-21T21:59:55.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.685+0000 7f9ddb5337c0 20 remove_watcher() i=0 2024-08-21T21:59:55.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.685+0000 7f9ddb5337c0 2 removed watcher, disabling cache 2024-08-21T21:59:55.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.686+0000 7f9ddb5337c0 20 remove_watcher() i=1 2024-08-21T21:59:55.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.687+0000 7f9ddb5337c0 20 remove_watcher() i=2 2024-08-21T21:59:55.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.688+0000 7f9ddb5337c0 20 remove_watcher() i=3 2024-08-21T21:59:55.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.689+0000 7f9ddb5337c0 20 remove_watcher() i=4 2024-08-21T21:59:55.691 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.690+0000 7f9ddb5337c0 20 remove_watcher() i=5 2024-08-21T21:59:55.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.691+0000 7f9ddb5337c0 20 remove_watcher() i=6 2024-08-21T21:59:55.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.692+0000 7f9ddb5337c0 20 remove_watcher() i=7 2024-08-21T21:59:55.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.692+0000 7f9daa7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:55.697 INFO:teuthology.orchestra.run.smithi178.stdout:[] 2024-08-21T21:59:55.713 DEBUG:tasks.util.rgw: json result: [] 2024-08-21T21:59:55.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:55.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:55.714 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-21T21:59:55.763 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:55.763 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:55.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.769+0000 7f179a0087c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:55.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.769+0000 7f179a0087c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:55.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.773+0000 7f17297fa640 20 reqs_thread_entry: start 2024-08-21T21:59:55.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.773+0000 7f179a0087c0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-08-21T21:59:55.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.773+0000 7f179a0087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.774+0000 7f179a0087c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.774+0000 7f179a0087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.776 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.774+0000 7f179a0087c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:55.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.781+0000 7f179a0087c0 20 RGWRados::pool_iterate: got zone_info.b607c794-1dc4-42ec-9ba3-bfb1b2f779e7 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.781+0000 7f179a0087c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.781+0000 7f179a0087c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.781+0000 7f179a0087c0 20 RGWRados::pool_iterate: got zonegroup_info.664b2903-5133-4ce7-b8a9-22c479b12fdd 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.781+0000 7f179a0087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.782+0000 7f179a0087c0 20 rados_obj.operate() r=0 bl.length=874 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.782+0000 7f179a0087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.782+0000 7f179a0087c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.782+0000 7f179a0087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.782+0000 7f179a0087c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.782+0000 7f179a0087c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.782+0000 7f179a0087c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "b607c794-1dc4-42ec-9ba3-bfb1b2f779e7", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "default", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "domain_root": "default.rgw.meta:root", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "control_pool": "default.rgw.control", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "log_pool": "default.rgw.log", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-21T21:59:55.788 INFO:teuthology.orchestra.run.smithi178.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "otp_pool": "default.rgw.otp", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "system_key": { 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "access_key": "", 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "secret_key": "" 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_pools": [ 2024-08-21T21:59:55.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "default-placement", 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": { 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD": { 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: "compression_type": "random" 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T21:59:55.790 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T21:59:55.791 INFO:teuthology.orchestra.run.smithi178.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-21T21:59:55.794 INFO:teuthology.orchestra.run.smithi178.stdout: "index_type": 0, 2024-08-21T21:59:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "inline_data": "true" 2024-08-21T21:59:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T21:59:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T21:59:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T21:59:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "", 2024-08-21T21:59:55.795 INFO:teuthology.orchestra.run.smithi178.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-21T21:59:55.795 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T21:59:55.796 DEBUG:tasks.util.rgw: json result: {'id': 'b607c794-1dc4-42ec-9ba3-bfb1b2f779e7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-21T21:59:55.797 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2024-08-21T21:59:55.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2024-08-21T21:59:55.797 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2024-08-21T21:59:55.836 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:55.836 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:55.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.842+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.843+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.843+0000 7f316721a7c0 20 realm 2024-08-21T21:59:55.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.843+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.843+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.843+0000 7f316721a7c0 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-08-21T21:59:55.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.843+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.844+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.844+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.844+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.844+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.844+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:55.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.845+0000 7f316721a7c0 20 searching for the correct realm 2024-08-21T21:59:55.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zone_info.b607c794-1dc4-42ec-9ba3-bfb1b2f779e7 2024-08-21T21:59:55.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:55.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zonegroup_info.664b2903-5133-4ce7-b8a9-22c479b12fdd 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.851+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.852+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.852+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.852+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.852+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.852+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.852+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.852+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zone_info.b607c794-1dc4-42ec-9ba3-bfb1b2f779e7 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 RGWRados::pool_iterate: got zonegroup_info.664b2903-5133-4ce7-b8a9-22c479b12fdd 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.858+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 zone default found 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 4 Realm: () 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 4 ZoneGroup: default (664b2903-5133-4ce7-b8a9-22c479b12fdd) 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 4 Zone: default (b607c794-1dc4-42ec-9ba3-bfb1b2f779e7) 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 10 cannot find current period zonegroup using local zonegroup configuration 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 zonegroup default 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 started sync module instance, tier type = 2024-08-21T21:59:55.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.859+0000 7f316721a7c0 20 started zone id=b607c794-1dc4-42ec-9ba3-bfb1b2f779e7 (name=default) with tier type = 2024-08-21T21:59:55.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.866+0000 7f316721a7c0 20 add_watcher() i=0 2024-08-21T21:59:55.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.866+0000 7f316721a7c0 20 add_watcher() i=1 2024-08-21T21:59:55.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.866+0000 7f316721a7c0 20 add_watcher() i=2 2024-08-21T21:59:55.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.866+0000 7f316721a7c0 20 add_watcher() i=3 2024-08-21T21:59:55.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.866+0000 7f316721a7c0 20 add_watcher() i=4 2024-08-21T21:59:55.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.866+0000 7f316721a7c0 20 add_watcher() i=5 2024-08-21T21:59:55.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.866+0000 7f316721a7c0 20 add_watcher() i=6 2024-08-21T21:59:55.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.867+0000 7f316721a7c0 20 add_watcher() i=7 2024-08-21T21:59:55.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.867+0000 7f316721a7c0 2 all 8 watchers are set, enabling cache 2024-08-21T21:59:55.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.868+0000 7f3133fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:55.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.868+0000 7f316721a7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:55.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.868+0000 7f316721a7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:55.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.872+0000 7f316721a7c0 5 note: GC not initialized 2024-08-21T21:59:55.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.872+0000 7f30a2ffd640 20 reqs_thread_entry: start 2024-08-21T21:59:55.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.873+0000 7f30a0ff9640 20 reqs_thread_entry: start 2024-08-21T21:59:55.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.873+0000 7f316721a7c0 20 init_complete bucket index max shards: 11 2024-08-21T21:59:55.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.873+0000 7f316721a7c0 10 rgw notify: Started notification manager with: 1 workers 2024-08-21T21:59:55.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.873+0000 7f3096ffd640 20 rgw notify: INFO: next queues processing will happen at: Wed Aug 21 22:00:26 2024 2024-08-21T21:59:55.875 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-08-21T21:59:55.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.873+0000 7f30967fc640 20 reqs_thread_entry: start 2024-08-21T21:59:55.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.881+0000 7f316721a7c0 20 remove_watcher() i=0 2024-08-21T21:59:55.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.881+0000 7f316721a7c0 2 removed watcher, disabling cache 2024-08-21T21:59:55.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.882+0000 7f316721a7c0 20 remove_watcher() i=1 2024-08-21T21:59:55.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.882+0000 7f316721a7c0 20 remove_watcher() i=2 2024-08-21T21:59:55.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.883+0000 7f316721a7c0 20 remove_watcher() i=3 2024-08-21T21:59:55.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.884+0000 7f316721a7c0 20 remove_watcher() i=4 2024-08-21T21:59:55.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.885+0000 7f316721a7c0 20 remove_watcher() i=5 2024-08-21T21:59:55.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.885+0000 7f316721a7c0 20 remove_watcher() i=6 2024-08-21T21:59:55.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.886+0000 7f316721a7c0 20 remove_watcher() i=7 2024-08-21T21:59:55.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.886+0000 7f3133fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-08-21T21:59:55.891 INFO:teuthology.orchestra.run.smithi178.stdout:[] 2024-08-21T21:59:55.893 DEBUG:tasks.util.rgw: json result: [] 2024-08-21T21:59:55.893 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:55.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2024-08-21T21:59:55.893 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2024-08-21T21:59:55.979 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:55.979 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:55.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.985+0000 7f5233d2c7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-08-21T21:59:55.986 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.985+0000 7f5233d2c7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-08-21T21:59:55.989 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.987+0000 7f51c97fa640 20 reqs_thread_entry: start 2024-08-21T21:59:55.989 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.987+0000 7f5233d2c7c0 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2024-08-21T21:59:55.989 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.987+0000 7f5233d2c7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.988+0000 7f5233d2c7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.988+0000 7f5233d2c7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.989+0000 7f5233d2c7c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 RGWRados::pool_iterate: got zone_info.b607c794-1dc4-42ec-9ba3-bfb1b2f779e7 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 RGWRados::pool_iterate: got zonegroup_info.664b2903-5133-4ce7-b8a9-22c479b12fdd 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 rados_obj.operate() r=0 bl.length=890 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-08-21T21:59:55.997 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.995+0000 7f5233d2c7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.996+0000 7f5233d2c7c0 20 rados_obj.operate() r=0 bl.length=46 2024-08-21T21:59:55.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.996+0000 7f5233d2c7c0 20 rados->read ofs=0 len=0 2024-08-21T21:59:55.998 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T21:59:55.996+0000 7f5233d2c7c0 20 rados_obj.operate() r=0 bl.length=362 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "b607c794-1dc4-42ec-9ba3-bfb1b2f779e7", 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "default", 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout: "domain_root": "default.rgw.meta:root", 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout: "control_pool": "default.rgw.control", 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout: "gc_pool": "default.rgw.log:gc", 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout: "lc_pool": "default.rgw.log:lc", 2024-08-21T21:59:56.003 INFO:teuthology.orchestra.run.smithi178.stdout: "log_pool": "default.rgw.log", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "intent_log_pool": "default.rgw.log:intent", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "usage_log_pool": "default.rgw.log:usage", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "roles_pool": "default.rgw.meta:roles", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "reshard_pool": "default.rgw.log:reshard", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "user_email_pool": "default.rgw.meta:users.email", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "otp_pool": "default.rgw.otp", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "system_key": { 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "access_key": "", 2024-08-21T21:59:56.004 INFO:teuthology.orchestra.run.smithi178.stdout: "secret_key": "" 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_pools": [ 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "default-placement", 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "index_pool": "default.rgw.buckets.index", 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": { 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD": { 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "data_pool": "default.rgw.buckets.data", 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: "compression_type": "random" 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T21:59:56.005 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: "index_type": 0, 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: "inline_data": "true" 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "", 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout: "notif_pool": "default.rgw.log:notif" 2024-08-21T21:59:56.006 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T21:59:56.006 DEBUG:tasks.util.rgw: json result: {'id': 'b607c794-1dc4-42ec-9ba3-bfb1b2f779e7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2024-08-21T21:59:56.007 INFO:tasks.rgw:Starting rgw... 2024-08-21T21:59:56.007 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2024-08-21T21:59:56.007 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-21T21:59:56.007 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-08-21T21:59:56.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-08-21T21:59:56.010 INFO:tasks.rgw.c1.client.0:Started 2024-08-21T21:59:56.011 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2024-08-21T21:59:56.011 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-21T21:59:56.011 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-21T21:59:56.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-21T21:59:56.013 INFO:tasks.rgw.c1.client.1:Started 2024-08-21T21:59:56.013 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2024-08-21T21:59:56.013 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-21T21:59:56.014 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-21T21:59:56.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-21T21:59:56.016 INFO:tasks.rgw.c2.client.0:Started 2024-08-21T21:59:56.016 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2024-08-21T21:59:56.017 INFO:tasks.rgw:Using beast as radosgw frontend 2024-08-21T21:59:56.017 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-08-21T21:59:56.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-08-21T21:59:56.019 INFO:tasks.rgw.c2.client.1:Started 2024-08-21T21:59:56.019 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi116.front.sepia.ceph.com:8000/ 2024-08-21T21:59:56.019 DEBUG:teuthology.orchestra.run.smithi116:> curl http://smithi116.front.sepia.ceph.com:8000/ 2024-08-21T21:59:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-21T21:59:56.055 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T21:59:56.055 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2024-08-21T21:59:56.055 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-21T21:59:56.055 INFO:teuthology.orchestra.run.smithi116.stderr:curl: (7) Failed to connect to smithi116.front.sepia.ceph.com port 8000: Connection refused 2024-08-21T21:59:57.056 DEBUG:teuthology.orchestra.run.smithi116:> curl http://smithi116.front.sepia.ceph.com:8000/ 2024-08-21T21:59:57.075 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T21:59:57.075 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2024-08-21T21:59:57.076 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-21T21:59:57.077 INFO:teuthology.orchestra.run.smithi116.stdout:anonymous 2024-08-21T21:59:57.077 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi116.front.sepia.ceph.com:8001/ 2024-08-21T21:59:57.077 DEBUG:teuthology.orchestra.run.smithi116:> curl http://smithi116.front.sepia.ceph.com:8001/ 2024-08-21T21:59:57.136 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T21:59:57.137 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2024-08-21T21:59:57.138 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-21T21:59:57.139 INFO:teuthology.orchestra.run.smithi116.stdout:anonymous 2024-08-21T21:59:57.139 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi178.front.sepia.ceph.com:8000/ 2024-08-21T21:59:57.140 DEBUG:teuthology.orchestra.run.smithi178:> curl http://smithi178.front.sepia.ceph.com:8000/ 2024-08-21T21:59:57.417 INFO:teuthology.orchestra.run.smithi178.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T21:59:57.417 INFO:teuthology.orchestra.run.smithi178.stderr: Dload Upload Total Spent Left Speed 2024-08-21T21:59:57.420 INFO:teuthology.orchestra.run.smithi178.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-08-21T21:59:57.420 INFO:teuthology.orchestra.run.smithi178.stdout:anonymous 2024-08-21T21:59:57.420 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi178.front.sepia.ceph.com:8001/ 2024-08-21T21:59:57.420 DEBUG:teuthology.orchestra.run.smithi178:> curl http://smithi178.front.sepia.ceph.com:8001/ 2024-08-21T21:59:57.441 INFO:teuthology.orchestra.run.smithi178.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T21:59:57.441 INFO:teuthology.orchestra.run.smithi178.stderr: Dload Upload Total Spent Left Speed 2024-08-21T21:59:57.442 INFO:teuthology.orchestra.run.smithi178.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2024-08-21T21:59:57.443 INFO:teuthology.orchestra.run.smithi178.stdout:anonymous 2024-08-21T21:59:57.443 INFO:teuthology.run_tasks:Running task rgw-multisite... 2024-08-21T21:59:57.510 INFO:tasks.rgw_multisite:creating realm.. 2024-08-21T21:59:57.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:57.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:57.510 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T21:59:57.551 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:57.551 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:57.581 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T21:59:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T21:59:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test-realm", 2024-08-21T21:59:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "current_period": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T21:59:57.582 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1 2024-08-21T21:59:57.582 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T21:59:57.583 INFO:tasks.rgw_multisite:creating master zonegroup.. 2024-08-21T21:59:57.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:57.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:57.584 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T21:59:57.666 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:57.666 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:57.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:57.677+0000 7f4134f377c0 -1 old format 2024-08-21T21:59:57.685 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T21:59:57.685 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a", 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "", 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [], 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [], 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T21:59:57.686 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T21:59:57.687 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T21:59:57.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:57.687 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T21:59:57.687 INFO:tasks.rgw_multisite:creating master zone.. 2024-08-21T21:59:57.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi116.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:57.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi116.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:57.688 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --endpoints http://smithi116.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T21:59:57.778 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:57.778 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:57.806 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T21:59:57.806 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T21:59:57.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a1", 2024-08-21T21:59:57.806 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "a1.rgw.meta:root", 2024-08-21T21:59:57.806 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "a1.rgw.control", 2024-08-21T21:59:57.806 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "a1.rgw.log:gc", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "a1.rgw.log:lc", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "a1.rgw.log", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-08-21T21:59:57.807 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "a1.rgw.otp", 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "a1.rgw.buckets.index", 2024-08-21T21:59:57.808 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T21:59:57.809 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T21:59:57.809 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "a1.rgw.buckets.data" 2024-08-21T21:59:57.809 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "a1.rgw.log:notif" 2024-08-21T21:59:57.810 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T21:59:57.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2024-08-21T21:59:58.190 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'a1.rgw.buckets.data' created 2024-08-21T21:59:58.212 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2024-08-21T21:59:59.193 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2024-08-21T21:59:59.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:59.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:59.231 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T21:59:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:59.305 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a1", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "a1.rgw.meta:root", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "a1.rgw.control", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "a1.rgw.log:gc", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "a1.rgw.log:lc", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "a1.rgw.log", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "a1.rgw.log:intent", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "a1.rgw.log:usage", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "a1.rgw.meta:roles", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "a1.rgw.log:reshard", 2024-08-21T21:59:59.306 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "a1.rgw.otp", 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T21:59:59.307 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "a1.rgw.buckets.index", 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "a1.rgw.buckets.data", 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: "compression_type": "random" 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:59.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "", 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "a1.rgw.log:notif" 2024-08-21T21:59:59.309 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T21:59:59.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:59.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T21:59:59.311 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T21:59:59.353 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T21:59:59.353 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T21:59:59.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T21:59:59.365+0000 7f6e1e51f7c0 0 period (c9075436-66f3-4d8c-a9e4-e295f6fc0789 does not have zone 5407971e-5f7f-4ba8-8316-03f497e31665 configured 2024-08-21T22:00:06.288 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 1, 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "predecessor_uuid": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": [], 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "period_map": { 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "zonegroups": [ 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a", 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T22:00:06.289 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a1", 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:00:06.290 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "false", 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T22:00:06.291 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:06.292 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: "short_zone_ids": [ 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 943360455 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zonegroup": "a", 2024-08-21T22:00:06.293 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "period_config": { 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:00:06.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "user_ratelimit": { 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_ratelimit": { 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:00:06.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "anonymous_ratelimit": { 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_name": "test-realm", 2024-08-21T22:00:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:00:06.297 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:00:06.298 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-08-21T22:00:06.298 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T22:00:06.340 INFO:tasks.rgw.c1.client.0.smithi116.stdout:2024-08-21T22:00:06.338+0000 7ff0d9250640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 24589) UID: 0 2024-08-21T22:00:06.340 INFO:tasks.rgw.c1.client.0.smithi116.stdout:2024-08-21T22:00:06.338+0000 7ff0dd7587c0 -1 shutting down 2024-08-21T22:00:06.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-08-21T22:00:12.400 INFO:tasks.rgw.c1.client.0:Stopped 2024-08-21T22:00:12.400 INFO:tasks.rgw.c1.client.0:Restarting daemon 2024-08-21T22:00:12.401 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2024-08-21T22:00:12.404 INFO:tasks.rgw.c1.client.0:Started 2024-08-21T22:00:12.404 DEBUG:teuthology.orchestra.run.smithi116:> curl http://smithi116.front.sepia.ceph.com:8000 2024-08-21T22:00:12.425 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:00:12.425 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:00:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-21T22:00:12.427 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-21T22:00:12.427 INFO:teuthology.orchestra.run.smithi116.stderr:curl: (7) Failed to connect to smithi116.front.sepia.ceph.com port 8000: Connection refused 2024-08-21T22:00:12.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2024-08-21T22:00:13.428 DEBUG:teuthology.orchestra.run.smithi116:> curl http://smithi116.front.sepia.ceph.com:8000 2024-08-21T22:00:13.449 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:00:13.449 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:00:13.450 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-21T22:00:13.451 INFO:teuthology.orchestra.run.smithi116.stdout:anonymous 2024-08-21T22:00:13.451 INFO:tasks.rgw_multisite:creating admin user.. 2024-08-21T22:00:13.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:13.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:13.452 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:13.542 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: "user_id": "realm-admin", 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: "display_name": "Realm Admin", 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: "email": "", 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: "suspended": 0, 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max_buckets": 1000, 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: "subusers": [], 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: "keys": [ 2024-08-21T22:00:13.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "user": "realm-admin", 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "swift_keys": [], 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": [], 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "op_mask": "read, write, delete", 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "system": "true", 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "default_storage_class": "", 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_tags": [], 2024-08-21T22:00:13.579 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:13.580 INFO:teuthology.orchestra.run.smithi116.stdout: "temp_url_keys": [], 2024-08-21T22:00:13.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "rgw", 2024-08-21T22:00:13.581 INFO:teuthology.orchestra.run.smithi116.stdout: "mfa_ids": [] 2024-08-21T22:00:13.581 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:00:13.581 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T22:00:13.589 INFO:tasks.rgw_multisite:creating zone.. 2024-08-21T22:00:13.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi116.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:13.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi116.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:13.590 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --endpoints http://smithi116.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:13.667 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a2", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "a2.rgw.meta:root", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "a2.rgw.control", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "a2.rgw.log:gc", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "a2.rgw.log:lc", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "a2.rgw.log", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-08-21T22:00:13.668 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "a2.rgw.otp", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T22:00:13.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "a2.rgw.buckets.index", 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "a2.rgw.buckets.data" 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T22:00:13.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:13.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:13.671 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:13.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:13.671 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "a2.rgw.log:notif" 2024-08-21T22:00:13.671 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:00:13.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2024-08-21T22:00:14.238 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'a2.rgw.buckets.data' created 2024-08-21T22:00:14.258 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2024-08-21T22:00:15.239 INFO:teuthology.orchestra.run.smithi116.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2024-08-21T22:00:15.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:15.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:15.269 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:15.346 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:00:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a2", 2024-08-21T22:00:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "a2.rgw.meta:root", 2024-08-21T22:00:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "a2.rgw.control", 2024-08-21T22:00:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "a2.rgw.log:gc", 2024-08-21T22:00:15.346 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "a2.rgw.log:lc", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "a2.rgw.log", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "a2.rgw.otp", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:00:15.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "a2.rgw.buckets.index", 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "a2.rgw.buckets.data", 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "compression_type": "random" 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T22:00:15.348 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T22:00:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "", 2024-08-21T22:00:15.349 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "a2.rgw.log:notif" 2024-08-21T22:00:15.349 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:00:15.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:15.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:15.351 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:15.393 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:15.394 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T22:00:15.401+0000 7fd70080f7c0 0 period (2584ad6b-0698-496d-8897-87f74ca5f014 does not have zone 3ccc7c5a-b8fe-4a58-84f4-d904ee91d189 configured 2024-08-21T22:00:18.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2024-08-21T22:00:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:Sending period to new master zone 5407971e-5f7f-4ba8-8316-03f497e31665 2024-08-21T22:00:22.342 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:00:22.342 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:22.342 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 2, 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "predecessor_uuid": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": [], 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "period_map": { 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "zonegroups": [ 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a", 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:22.343 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a2", 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8001" 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T22:00:22.344 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a1", 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T22:00:22.345 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T22:00:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "short_zone_ids": [ 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:22.347 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 4021227367 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 943360455 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zonegroup": "a", 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "period_config": { 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:00:22.348 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "user_ratelimit": { 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:00:22.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_ratelimit": { 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "anonymous_ratelimit": { 2024-08-21T22:00:22.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_name": "test-realm", 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:00:22.351 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:00:22.352 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2024-08-21T22:00:22.353 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T22:00:22.394 INFO:tasks.rgw.c1.client.1.smithi116.stdout:2024-08-21T22:00:22.392+0000 7f6306917640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 24590) UID: 0 2024-08-21T22:00:22.394 INFO:tasks.rgw.c1.client.1.smithi116.stdout:2024-08-21T22:00:22.392+0000 7f630cf217c0 -1 shutting down 2024-08-21T22:00:23.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-21T22:00:28.454 INFO:tasks.rgw.c1.client.1:Stopped 2024-08-21T22:00:28.455 INFO:tasks.rgw.c1.client.1:Restarting daemon 2024-08-21T22:00:28.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2024-08-21T22:00:28.458 INFO:tasks.rgw.c1.client.1:Started 2024-08-21T22:00:28.458 DEBUG:teuthology.orchestra.run.smithi116:> curl http://smithi116.front.sepia.ceph.com:8001 2024-08-21T22:00:28.481 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-21T22:00:28.481 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:00:28.481 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:00:28.481 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-21T22:00:28.481 INFO:teuthology.orchestra.run.smithi116.stderr:curl: (7) Failed to connect to smithi116.front.sepia.ceph.com port 8001: Connection refused 2024-08-21T22:00:29.482 DEBUG:teuthology.orchestra.run.smithi116:> curl http://smithi116.front.sepia.ceph.com:8001 2024-08-21T22:00:29.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2024-08-21T22:00:29.503 INFO:teuthology.orchestra.run.smithi116.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:00:29.503 INFO:teuthology.orchestra.run.smithi116.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:00:29.505 INFO:teuthology.orchestra.run.smithi116.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-21T22:00:29.505 INFO:teuthology.orchestra.run.smithi116.stdout:anonymous 2024-08-21T22:00:29.505 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-08-21T22:00:29.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi116.front.sepia.ceph.com:8000', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:29.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi116.front.sepia.ceph.com:8000', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:29.506 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi116.front.sepia.ceph.com:8000 --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:29.559 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:29.559 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:29.599 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:29.600 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:29.600 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "test-realm", 2024-08-21T22:00:29.600 INFO:teuthology.orchestra.run.smithi178.stdout: "current_period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:29.600 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 2 2024-08-21T22:00:29.600 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:29.601 INFO:tasks.rgw_multisite:creating zonegroup.. 2024-08-21T22:00:29.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:29.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:29.601 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:29.647 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:29.647 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:29.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T22:00:29.659+0000 7f6b4dd257c0 -1 old format 2024-08-21T22:00:29.666 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:29.666 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "b", 2024-08-21T22:00:29.666 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b", 2024-08-21T22:00:29.666 INFO:teuthology.orchestra.run.smithi178.stdout: "api_name": "", 2024-08-21T22:00:29.666 INFO:teuthology.orchestra.run.smithi178.stdout: "is_master": "false", 2024-08-21T22:00:29.666 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:29.666 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames": [], 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames_s3website": [], 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zone": "", 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "zones": [], 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_targets": [], 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "default_placement": "", 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:29.667 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_policy": { 2024-08-21T22:00:29.668 INFO:teuthology.orchestra.run.smithi178.stdout: "groups": [] 2024-08-21T22:00:29.668 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:29.668 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:29.669 INFO:tasks.rgw_multisite:creating zone.. 2024-08-21T22:00:29.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi178.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:29.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi178.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:29.669 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --endpoints http://smithi178.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:29.758 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:29.758 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:29.786 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:29.786 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:29.786 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b1", 2024-08-21T22:00:29.786 INFO:teuthology.orchestra.run.smithi178.stdout: "domain_root": "b1.rgw.meta:root", 2024-08-21T22:00:29.786 INFO:teuthology.orchestra.run.smithi178.stdout: "control_pool": "b1.rgw.control", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "gc_pool": "b1.rgw.log:gc", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "lc_pool": "b1.rgw.log:lc", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "log_pool": "b1.rgw.log", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-08-21T22:00:29.787 INFO:teuthology.orchestra.run.smithi178.stdout: "otp_pool": "b1.rgw.otp", 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "system_key": { 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_pools": [ 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "default-placement", 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "index_pool": "b1.rgw.buckets.index", 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": { 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD": { 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: "data_pool": "b1.rgw.buckets.data" 2024-08-21T22:00:29.788 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: "index_type": 0, 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: "inline_data": "true" 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout: "notif_pool": "b1.rgw.log:notif" 2024-08-21T22:00:29.789 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:29.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2024-08-21T22:00:30.697 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'b1.rgw.buckets.data' created 2024-08-21T22:00:30.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2024-08-21T22:00:31.700 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2024-08-21T22:00:31.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:31.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:31.728 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:31.780 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:31.780 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:31.805 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b1", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "domain_root": "b1.rgw.meta:root", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "control_pool": "b1.rgw.control", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "gc_pool": "b1.rgw.log:gc", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "lc_pool": "b1.rgw.log:lc", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "log_pool": "b1.rgw.log", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "intent_log_pool": "b1.rgw.log:intent", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "usage_log_pool": "b1.rgw.log:usage", 2024-08-21T22:00:31.806 INFO:teuthology.orchestra.run.smithi178.stdout: "roles_pool": "b1.rgw.meta:roles", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "reshard_pool": "b1.rgw.log:reshard", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "otp_pool": "b1.rgw.otp", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "system_key": { 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_pools": [ 2024-08-21T22:00:31.807 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "default-placement", 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "index_pool": "b1.rgw.buckets.index", 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": { 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD": { 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "data_pool": "b1.rgw.buckets.data", 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "compression_type": "random" 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "index_type": 0, 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: "inline_data": "true" 2024-08-21T22:00:31.808 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:31.809 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:31.809 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:31.809 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "", 2024-08-21T22:00:31.809 INFO:teuthology.orchestra.run.smithi178.stdout: "notif_pool": "b1.rgw.log:notif" 2024-08-21T22:00:31.809 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:31.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:31.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:31.810 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:31.853 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:31.853 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:31.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T22:00:31.862+0000 7fe3569307c0 0 period (2584ad6b-0698-496d-8897-87f74ca5f014 does not have zone fe48b235-c997-4b77-a96a-2d49ab4ea593 configured 2024-08-21T22:00:38.758 INFO:teuthology.orchestra.run.smithi178.stderr:Sending period to new master zone 5407971e-5f7f-4ba8-8316-03f497e31665 2024-08-21T22:00:38.788 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:38.788 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:38.788 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 3, 2024-08-21T22:00:38.788 INFO:teuthology.orchestra.run.smithi178.stdout: "predecessor_uuid": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_status": [], 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "period_map": { 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "zonegroups": [ 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "a", 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "a", 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "api_name": "", 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "is_master": "true", 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.789 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames": [], 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames_s3website": [], 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "zones": [ 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "a2", 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi116.front.sepia.ceph.com:8001" 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "log_meta": "false", 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "log_data": "true", 2024-08-21T22:00:38.790 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "read_only": "false", 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_type": "", 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from_all": "true", 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from": [], 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "redirect_zone": "" 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "a1", 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "log_meta": "false", 2024-08-21T22:00:38.791 INFO:teuthology.orchestra.run.smithi178.stdout: "log_data": "true", 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "read_only": "false", 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_type": "", 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from_all": "true", 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from": [], 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "redirect_zone": "" 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_targets": [ 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "default-placement", 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "tags": [], 2024-08-21T22:00:38.792 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": [ 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD" 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: "default_placement": "default-placement", 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_policy": { 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: "groups": [] 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "b", 2024-08-21T22:00:38.793 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b", 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "api_name": "", 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "is_master": "false", 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames": [], 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames_s3website": [], 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zone": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "zones": [ 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b1", 2024-08-21T22:00:38.794 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "log_meta": "false", 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "log_data": "false", 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "read_only": "false", 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_type": "", 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from_all": "true", 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from": [], 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: "redirect_zone": "" 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.795 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_targets": [ 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "default-placement", 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: "tags": [], 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": [ 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD" 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: "default_placement": "default-placement", 2024-08-21T22:00:38.796 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_policy": { 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: "groups": [] 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: "short_zone_ids": [ 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:38.803 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 4021227367 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 943360455 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 3475667924 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zonegroup": "a", 2024-08-21T22:00:38.804 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: "period_config": { 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_quota": { 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false, 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: "check_on_raw": false, 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size": -1, 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size_kb": 0, 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: "max_objects": -1 2024-08-21T22:00:38.805 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "user_quota": { 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false, 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "check_on_raw": false, 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size": -1, 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size_kb": 0, 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "max_objects": -1 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "user_ratelimit": { 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_ops": 0, 2024-08-21T22:00:38.806 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_ops": 0, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_bytes": 0, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_bytes": 0, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_ratelimit": { 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_ops": 0, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_ops": 0, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_bytes": 0, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_bytes": 0, 2024-08-21T22:00:38.807 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "anonymous_ratelimit": { 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_ops": 0, 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_ops": 0, 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_bytes": 0, 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_bytes": 0, 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_name": "test-realm", 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:00:38.808 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:38.809 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2024-08-21T22:00:38.809 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T22:00:38.845 INFO:tasks.rgw.c2.client.0.smithi178.stdout:2024-08-21T22:00:38.843+0000 7ff318e50640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 24595) UID: 0 2024-08-21T22:00:38.845 INFO:tasks.rgw.c2.client.0.smithi178.stdout:2024-08-21T22:00:38.843+0000 7ff31d3587c0 -1 shutting down 2024-08-21T22:00:39.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-21T22:00:44.911 INFO:tasks.rgw.c2.client.0:Stopped 2024-08-21T22:00:44.911 INFO:tasks.rgw.c2.client.0:Restarting daemon 2024-08-21T22:00:44.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2024-08-21T22:00:44.915 INFO:tasks.rgw.c2.client.0:Started 2024-08-21T22:00:44.915 DEBUG:teuthology.orchestra.run.smithi178:> curl http://smithi178.front.sepia.ceph.com:8000 2024-08-21T22:00:44.936 INFO:teuthology.orchestra.run.smithi178.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:00:44.936 INFO:teuthology.orchestra.run.smithi178.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:00:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-21T22:00:44.938 INFO:teuthology.orchestra.run.smithi178.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-21T22:00:44.938 INFO:teuthology.orchestra.run.smithi178.stderr:curl: (7) Failed to connect to smithi178.front.sepia.ceph.com port 8000: Connection refused 2024-08-21T22:00:45.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2024-08-21T22:00:45.939 DEBUG:teuthology.orchestra.run.smithi178:> curl http://smithi178.front.sepia.ceph.com:8000 2024-08-21T22:00:45.958 INFO:teuthology.orchestra.run.smithi178.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:00:45.958 INFO:teuthology.orchestra.run.smithi178.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:00:45.960 INFO:teuthology.orchestra.run.smithi178.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-21T22:00:45.961 INFO:teuthology.orchestra.run.smithi178.stdout:anonymous 2024-08-21T22:00:45.961 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2024-08-21T22:00:45.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi116.front.sepia.ceph.com:8000', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--url', 'http://smithi116.front.sepia.ceph.com:8000', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:45.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi116.front.sepia.ceph.com:8000', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--url', 'http://smithi116.front.sepia.ceph.com:8000', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:45.961 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi116.front.sepia.ceph.com:8000 --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --url http://smithi116.front.sepia.ceph.com:8000 --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:46.053 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:46.053 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:46.082 INFO:tasks.rgw.c2.client.0.smithi178.stdout:2024-08-21T22:00:46.079+0000 7f1e67dff640 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.c5b47b13-9a2b-401b-92e0-5cba73a5c237.control err=-107 2024-08-21T22:00:46.091 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:46.091 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:46.091 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "test-realm", 2024-08-21T22:00:46.091 INFO:teuthology.orchestra.run.smithi178.stdout: "current_period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:46.091 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 2 2024-08-21T22:00:46.091 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:46.092 INFO:tasks.rgw_multisite:creating zone.. 2024-08-21T22:00:46.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi178.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:46.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', 'SFQCSPIH466OOI28', '--secret', 'Y6NKkqiemixittNnTBCkGQzIGFnED8DL', '--endpoints', 'http://smithi178.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:46.093 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key SFQCSPIH466OOI28 --secret Y6NKkqiemixittNnTBCkGQzIGFnED8DL --endpoints http://smithi178.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:46.141 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:46.141 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:46.168 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:46.168 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T22:00:46.168 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b2", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "domain_root": "b2.rgw.meta:root", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "control_pool": "b2.rgw.control", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "gc_pool": "b2.rgw.log:gc", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "lc_pool": "b2.rgw.log:lc", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "log_pool": "b2.rgw.log", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-08-21T22:00:46.169 INFO:teuthology.orchestra.run.smithi178.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "otp_pool": "b2.rgw.otp", 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "system_key": { 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_pools": [ 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "default-placement", 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "index_pool": "b2.rgw.buckets.index", 2024-08-21T22:00:46.170 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": { 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD": { 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: "data_pool": "b2.rgw.buckets.data" 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: "index_type": 0, 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: "inline_data": "true" 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout: "notif_pool": "b2.rgw.log:notif" 2024-08-21T22:00:46.171 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:46.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2024-08-21T22:00:46.733 INFO:teuthology.orchestra.run.smithi178.stderr:pool 'b2.rgw.buckets.data' created 2024-08-21T22:00:46.763 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2024-08-21T22:00:47.735 INFO:teuthology.orchestra.run.smithi178.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2024-08-21T22:00:47.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:47.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:47.774 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:47.821 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:47.821 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:47.845 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:47.845 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T22:00:47.845 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b2", 2024-08-21T22:00:47.845 INFO:teuthology.orchestra.run.smithi178.stdout: "domain_root": "b2.rgw.meta:root", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "control_pool": "b2.rgw.control", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "gc_pool": "b2.rgw.log:gc", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "lc_pool": "b2.rgw.log:lc", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "log_pool": "b2.rgw.log", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "intent_log_pool": "b2.rgw.log:intent", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "usage_log_pool": "b2.rgw.log:usage", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "roles_pool": "b2.rgw.meta:roles", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "reshard_pool": "b2.rgw.log:reshard", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2024-08-21T22:00:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "otp_pool": "b2.rgw.otp", 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "system_key": { 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_pools": [ 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "default-placement", 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "index_pool": "b2.rgw.buckets.index", 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": { 2024-08-21T22:00:47.847 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD": { 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: "data_pool": "b2.rgw.buckets.data", 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: "compression_type": "random" 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: "index_type": 0, 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: "inline_data": "true" 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "", 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout: "notif_pool": "b2.rgw.log:notif" 2024-08-21T22:00:47.848 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:47.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:47.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:00:47.850 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2024-08-21T22:00:47.893 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:00:47.893 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:00:47.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-08-21T22:00:47.902+0000 7fb8f121b7c0 0 period (2584ad6b-0698-496d-8897-87f74ca5f014 does not have zone 11250536-b0eb-44b2-8efe-f0ea767669ff configured 2024-08-21T22:00:50.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2024-08-21T22:00:54.802 INFO:teuthology.orchestra.run.smithi178.stderr:Sending period to new master zone 5407971e-5f7f-4ba8-8316-03f497e31665 2024-08-21T22:00:54.833 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "epoch": 4, 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "predecessor_uuid": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_status": [], 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "period_map": { 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "zonegroups": [ 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "a", 2024-08-21T22:00:54.834 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "a", 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "api_name": "", 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "is_master": "true", 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames": [], 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames_s3website": [], 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "zones": [ 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "a2", 2024-08-21T22:00:54.835 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi116.front.sepia.ceph.com:8001" 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "log_meta": "false", 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "log_data": "true", 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "read_only": "false", 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_type": "", 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from_all": "true", 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from": [], 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "redirect_zone": "" 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.836 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "a1", 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "log_meta": "false", 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "log_data": "true", 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "read_only": "false", 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_type": "", 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from_all": "true", 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from": [], 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: "redirect_zone": "" 2024-08-21T22:00:54.837 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_targets": [ 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "default-placement", 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "tags": [], 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": [ 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD" 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "default_placement": "default-placement", 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:54.838 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_policy": { 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: "groups": [] 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "b", 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b", 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: "api_name": "", 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: "is_master": "false", 2024-08-21T22:00:54.839 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames": [], 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "hostnames_s3website": [], 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zone": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "zones": [ 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b2", 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi178.front.sepia.ceph.com:8001" 2024-08-21T22:00:54.840 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "log_meta": "false", 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "log_data": "true", 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "read_only": "false", 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_type": "", 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from_all": "true", 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from": [], 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "redirect_zone": "" 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "id": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:54.841 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "b1", 2024-08-21T22:00:54.842 INFO:teuthology.orchestra.run.smithi178.stdout: "endpoints": [ 2024-08-21T22:00:54.842 INFO:teuthology.orchestra.run.smithi178.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T22:00:54.842 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.842 INFO:teuthology.orchestra.run.smithi178.stdout: "log_meta": "false", 2024-08-21T22:00:54.842 INFO:teuthology.orchestra.run.smithi178.stdout: "log_data": "true", 2024-08-21T22:00:54.842 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:00:54.842 INFO:teuthology.orchestra.run.smithi178.stdout: "read_only": "false", 2024-08-21T22:00:54.845 INFO:teuthology.orchestra.run.smithi178.stdout: "tier_type": "", 2024-08-21T22:00:54.845 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from_all": "true", 2024-08-21T22:00:54.845 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_from": [], 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: "redirect_zone": "" 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: "placement_targets": [ 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: "name": "default-placement", 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: "tags": [], 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: "storage_classes": [ 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: "STANDARD" 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.846 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "default_placement": "default-placement", 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_policy": { 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "groups": [] 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: ], 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "short_zone_ids": [ 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 1441323143 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.847 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 4021227367 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 943360455 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: "key": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 3475667924 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.848 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zonegroup": "a", 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "period_config": { 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_quota": { 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "check_on_raw": false, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size": -1, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size_kb": 0, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "max_objects": -1 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "user_quota": { 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "check_on_raw": false, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size": -1, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "max_size_kb": 0, 2024-08-21T22:00:54.849 INFO:teuthology.orchestra.run.smithi178.stdout: "max_objects": -1 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "user_ratelimit": { 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_ops": 0, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_ops": 0, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_bytes": 0, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_bytes": 0, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "bucket_ratelimit": { 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_ops": 0, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_ops": 0, 2024-08-21T22:00:54.850 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_bytes": 0, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_bytes": 0, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "anonymous_ratelimit": { 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_ops": 0, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_ops": 0, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "max_read_bytes": 0, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "max_write_bytes": 0, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "enabled": false 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_name": "test-realm", 2024-08-21T22:00:54.851 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:00:54.852 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:00:54.853 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2024-08-21T22:00:54.853 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T22:00:54.853 INFO:tasks.rgw.c2.client.1.smithi178.stdout:2024-08-21T22:00:54.852+0000 7f509a7fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 24600) UID: 0 2024-08-21T22:00:54.854 INFO:tasks.rgw.c2.client.1.smithi178.stdout:2024-08-21T22:00:54.852+0000 7f50a33777c0 -1 shutting down 2024-08-21T22:00:56.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-08-21T22:01:00.955 INFO:tasks.rgw.c2.client.1:Stopped 2024-08-21T22:01:00.955 INFO:tasks.rgw.c2.client.1:Restarting daemon 2024-08-21T22:01:00.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2024-08-21T22:01:00.959 INFO:tasks.rgw.c2.client.1:Started 2024-08-21T22:01:00.959 DEBUG:teuthology.orchestra.run.smithi178:> curl http://smithi178.front.sepia.ceph.com:8001 2024-08-21T22:01:00.980 INFO:teuthology.orchestra.run.smithi178.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:01:00.980 INFO:teuthology.orchestra.run.smithi178.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:01:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-08-21T22:01:00.982 INFO:teuthology.orchestra.run.smithi178.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-08-21T22:01:00.982 INFO:teuthology.orchestra.run.smithi178.stderr:curl: (7) Failed to connect to smithi178.front.sepia.ceph.com port 8001: Connection refused 2024-08-21T22:01:01.983 DEBUG:teuthology.orchestra.run.smithi178:> curl http://smithi178.front.sepia.ceph.com:8001 2024-08-21T22:01:02.003 INFO:teuthology.orchestra.run.smithi178.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-08-21T22:01:02.003 INFO:teuthology.orchestra.run.smithi178.stderr: Dload Upload Total Spent Left Speed 2024-08-21T22:01:02.004 INFO:teuthology.orchestra.run.smithi178.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-08-21T22:01:02.006 INFO:teuthology.orchestra.run.smithi178.stdout:anonymous 2024-08-21T22:01:02.006 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2024-08-21T22:01:02.006 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2024-08-21T22:01:02.021 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi 2024-08-21T22:01:02.044 INFO:tasks.rgw_multisite_tests:creating test user.. 2024-08-21T22:01:02.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:01:02.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:01:02.044 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T22:01:02.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2024-08-21T22:01:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:02.101 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:02.149 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "user_id": "rgw-multisite-test-user", 2024-08-21T22:01:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "display_name": "Multisite Test User", 2024-08-21T22:01:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "email": "", 2024-08-21T22:01:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "suspended": 0, 2024-08-21T22:01:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max_buckets": 1000, 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "subusers": [], 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "keys": [ 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "user": "rgw-multisite-test-user", 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "1TZPTET2OKXOCZI6JXMG", 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "zoGU4u2TlyqsQ7GDjZFXVliE47Xs92TQMAQDZiRt" 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "swift_keys": [], 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": [ 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "roles", 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "perm": "*" 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "op_mask": "read, write, delete", 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_storage_class": "", 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_tags": [], 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:01:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "temp_url_keys": [], 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "rgw", 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "mfa_ids": [] 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-21T22:01:02.161 INFO:rgw_multi.tests:meta checkpoint 2024-08-21T22:01:02.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:02.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:02.162 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:02.205 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:02.205 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:01:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:01:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.251 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:01:02.252 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:01:02.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:02.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:02.252 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:02.296 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:02.297 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:02.344 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:02.354 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:02.356 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:07.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:07.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:07.357 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:07.407 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:07.458 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:07.458 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:07.458 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:07.459 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:07.460 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:07.461 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:07.462 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:07.463 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:07.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:07.464 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:07.465 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:12.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:12.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:12.466 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:12.517 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:12.517 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:12.567 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:12.567 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:12.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:12.569 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:12.570 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:12.571 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:12.572 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:12.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:12.573 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:12.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:17.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:17.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:17.575 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:17.625 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:17.625 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:17.675 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:17.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:17.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:17.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:17.679 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:17.680 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:17.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:22.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:22.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:22.682 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:22.733 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:22.733 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:22.784 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:22.788 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:22.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:22.790 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:22.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:27.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:27.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:27.792 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:27.843 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:27.892 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:27.893 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:27.894 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:27.895 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:27.896 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:27.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:27.897 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:27.897 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:27.897 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:27.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:27.897 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:27.898 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:32.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:32.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:32.899 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:32.950 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:32.950 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:33.001 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:33.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:33.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:33.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:33.005 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:33.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:33.007 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:33.009 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:38.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:38.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:38.010 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:38.060 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:38.060 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:38.110 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:38.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:38.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:38.113 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:38.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:38.115 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:38.116 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:43.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:43.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:43.117 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:43.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:43.220 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:43.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:43.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:43.223 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:43.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:43.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:43.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:48.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:48.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:48.226 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:48.279 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:48.280 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:48.335 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:48.335 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:48.335 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:48.335 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:48.335 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:48.335 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:48.335 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:48.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:48.337 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:48.340 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:48.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:53.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:53.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:53.342 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:53.439 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:53.439 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:53.440 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:53.440 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:53.440 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:53.440 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:53.440 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:53.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:53.440 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:53.441 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:53.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:53.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:53.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:53.445 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:53.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:01:58.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:58.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:01:58.447 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:01:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:01:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:01:58.550 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:01:58.550 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:01:58.550 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:01:58.550 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:01:58.550 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:01:58.550 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:01:58.550 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:58.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:58.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:58.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:58.554 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:58.555 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:01:58.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:01:58.555 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:01:58.555 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:01:58.555 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:01:58.555 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:01:58.555 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:01:58.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:03.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:03.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:03.557 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:03.609 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:03.660 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:03.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:03.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:03.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:03.665 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:03.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:08.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:08.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:08.667 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:08.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:08.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:08.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:08.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:08.775 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:08.776 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:13.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:13.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:13.777 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:13.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:13.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:13.878 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:13.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:13.880 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:13.881 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:13.882 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:13.883 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:18.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:18.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:18.885 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:18.937 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:18.987 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:18.988 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:18.989 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:18.990 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:18.991 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:18.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:18.993 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:18.993 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:18.993 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:18.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:18.993 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:18.994 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:23.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:23.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:23.995 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:24.046 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:24.098 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:24.099 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:24.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:24.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:24.103 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:24.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:24.103 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:24.103 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:24.103 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:24.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:24.103 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:24.104 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:29.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:29.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:29.105 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:29.156 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:29.206 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:29.206 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:29.207 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:29.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:29.209 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:29.210 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:29.211 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:29.212 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:34.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:34.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:34.213 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:34.264 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:34.265 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:34.319 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:34.320 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:39.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:39.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:39.322 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:39.371 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:39.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:39.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:39.421 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:39.422 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:39.423 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:39.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:39.424 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:39.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:44.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:44.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:44.426 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:44.477 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:44.478 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:44.528 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:44.529 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:44.530 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:44.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:44.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:44.533 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:44.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:44.534 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:44.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:44.534 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:44.535 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:49.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:49.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:49.536 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:49.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:49.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:49.642 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:49.643 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:54.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:54.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:54.645 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:54.696 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:54.696 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:54.748 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:54.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:54.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:54.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:54.753 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:54.753 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:54.754 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1724277662.145957_63.1 target= 2024-08-21T22:02:59.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:59.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:59.756 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:59.806 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:59.807 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:59.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:59.863 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:02:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:02:59.865 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:02:59.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:02:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:02:59.867 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-08-21T22:02:59.867 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-08-21T22:02:59.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:59.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:59.867 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:02:59.918 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:02:59.919 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_status": { 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout: "info": { 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout: "status": "sync", 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout: "num_shards": 4, 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:02:59.969 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "markers": [ 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 0, 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 1, 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:02:59.970 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 1, 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:02:59.971 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 2, 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:02:59.972 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 3, 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:02:59.973 INFO:teuthology.orchestra.run.smithi178.stdout: "full_sync": { 2024-08-21T22:02:59.974 INFO:teuthology.orchestra.run.smithi178.stdout: "total": 1, 2024-08-21T22:02:59.974 INFO:teuthology.orchestra.run.smithi178.stdout: "complete": 1 2024-08-21T22:02:59.974 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:02:59.974 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:02:59.975 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-08-21T22:02:59.975 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-08-21T22:02:59.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:59.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:02:59.975 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:00.020 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:00.020 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:00.065 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:03:00.065 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_status": { 2024-08-21T22:03:00.065 INFO:teuthology.orchestra.run.smithi178.stdout: "info": { 2024-08-21T22:03:00.065 INFO:teuthology.orchestra.run.smithi178.stdout: "status": "sync", 2024-08-21T22:03:00.065 INFO:teuthology.orchestra.run.smithi178.stdout: "num_shards": 4, 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "markers": [ 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 0, 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 1, 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:00.066 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 1, 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.067 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 2, 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.068 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 3, 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "full_sync": { 2024-08-21T22:03:00.069 INFO:teuthology.orchestra.run.smithi178.stdout: "total": 1, 2024-08-21T22:03:00.070 INFO:teuthology.orchestra.run.smithi178.stdout: "complete": 1 2024-08-21T22:03:00.070 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.070 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:03:00.071 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-08-21T22:03:00.071 INFO:rgw_multi.tests:meta checkpoint 2024-08-21T22:03:00.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.071 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:00.116 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:00.117 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:00.153 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.164 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:00.165 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:00.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.165 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.165 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:00.165 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:00.165 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:00.165 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:03:00.166 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:03:00.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.166 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:00.210 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:00.259 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:00.260 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:00.261 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:00.262 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-08-21T22:03:00.262 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2024-08-21T22:03:00.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.263 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:00.307 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:00.307 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_status": { 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout: "info": { 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout: "status": "sync", 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout: "num_shards": 4, 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.351 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "markers": [ 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 0, 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 1, 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.352 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 1, 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 2, 2024-08-21T22:03:00.353 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 3, 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.354 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "full_sync": { 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "total": 1, 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: "complete": 1 2024-08-21T22:03:00.355 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.356 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:03:00.356 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2024-08-21T22:03:00.357 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2024-08-21T22:03:00.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:00.357 DEBUG:teuthology.orchestra.run.smithi178:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:00.400 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:00.401 INFO:teuthology.orchestra.run.smithi178.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "sync_status": { 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "info": { 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "status": "sync", 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "num_shards": 4, 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "markers": [ 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.445 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 0, 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 1, 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.446 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 1, 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 2, 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.447 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: { 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "key": 3, 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "val": { 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "state": 1, 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "marker": "", 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "next_step_marker": "", 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "total_entries": 0, 2024-08-21T22:03:00.448 INFO:teuthology.orchestra.run.smithi178.stdout: "pos": 0, 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: "timestamp": "0.000000", 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: "realm_epoch": 2 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: "full_sync": { 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: "total": 1, 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: "complete": 1 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-08-21T22:03:00.449 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-08-21T22:03:00.450 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2024-08-21T22:03:00.450 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2024-08-21T22:03:00.460 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-1 2024-08-21T22:03:03.458 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-2 2024-08-21T22:03:06.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:06.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:06.471 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:06.516 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:06.517 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:06.552 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:03:06.563 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:03:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:03:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277783.477492_77.1", 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:03:03.477492Z" 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:06.564 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:03:06.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:03:06.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:06.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:06.566 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:06.609 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:06.660 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:06.660 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:06.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:06.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:06.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277783.442681_70.1", 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:03.442681Z", 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:06.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:06.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:06.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:06.666 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:06.666 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:06.666 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:06.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:06.666 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:06.668 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277783.477492_77.1 target=1_1724277783.442681_70.1 2024-08-21T22:03:11.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:11.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:11.669 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:11.774 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:11.774 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:11.774 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:11.775 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:11.776 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277783.477492_77.1", 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:11.777 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:03.477492Z", 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:11.778 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:11.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-08-21T22:03:11.789 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2024-08-21T22:03:11.789 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-3 2024-08-21T22:03:11.804 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-4 2024-08-21T22:03:11.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:11.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:11.825 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:11.867 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:11.901 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:03:11.912 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:03:11.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277791.815050_85.1", 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:03:11.815050Z" 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:11.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:11.914 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:11.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:11.914 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:03:11.915 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:03:11.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:11.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:11.915 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:12.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:12.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:12.053 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:12.053 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:12.053 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:12.053 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:12.053 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:12.053 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277791.815050_85.1", 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:11.815050Z", 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.058 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:12.059 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-08-21T22:03:12.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.132 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:12.174 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:12.208 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:03:12.218 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277791.815050_85.1", 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:03:11.815050Z" 2024-08-21T22:03:12.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.220 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:12.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.220 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:03:12.221 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:03:12.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.221 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:12.305 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:12.305 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:12.349 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:12.349 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:12.349 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:12.349 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:12.349 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:12.349 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:12.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.351 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277791.815050_85.1", 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:11.815050Z", 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:12.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.354 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:12.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-08-21T22:03:12.363 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2024-08-21T22:03:12.363 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-5 2024-08-21T22:03:12.380 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-6 2024-08-21T22:03:12.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.411 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:12.455 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:12.455 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:12.488 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277792.398034_93.1", 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:03:12.398034Z" 2024-08-21T22:03:12.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.500 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.500 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:12.500 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.500 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:03:12.501 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:03:12.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:12.501 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:12.544 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:12.545 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:12.594 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:12.594 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:12.594 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:12.594 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:12.594 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:12.594 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:12.594 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.596 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277792.377759_89.1", 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:12.377759Z", 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:12.597 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:12.598 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:12.599 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:12.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:12.599 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:12.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277792.398034_93.1 target=1_1724277792.377759_89.1 2024-08-21T22:03:17.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:17.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:17.601 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:17.651 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:17.652 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:17.702 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:17.702 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:17.702 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:17.702 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:17.703 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:17.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277792.398034_93.1", 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:17.705 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:12.398034Z", 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:17.706 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:17.707 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:17.708 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2024-08-21T22:03:17.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:17.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:17.802 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:17.858 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:17.903 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:03:17.914 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:03:17.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.914 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:17.914 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:03:17.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.768676_101.1", 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:03:17.768676Z" 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:03:17.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:17.916 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:03:17.917 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:03:17.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:17.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:17.917 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:17.960 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:17.960 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:18.004 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:18.004 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:18.005 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:18.006 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:18.007 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:18.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:18.009 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:18.010 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:23.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:23.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:23.015 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:23.065 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:23.066 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:23.115 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:23.115 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:23.115 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:23.115 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:23.116 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:23.117 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:23.119 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:23.120 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:23.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:23.121 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:23.122 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:28.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:28.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:28.124 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:28.175 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:28.175 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:28.227 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:28.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:28.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:28.230 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:28.231 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:28.232 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:28.233 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:33.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:33.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:33.234 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:33.280 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:33.281 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:33.327 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:33.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:33.329 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:33.331 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:33.332 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:33.332 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:33.332 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:33.333 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:38.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:38.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:38.334 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:38.382 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:38.383 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:38.429 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:38.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:38.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:38.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:38.433 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:38.434 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:38.435 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:43.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:43.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:43.437 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:43.490 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:43.490 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:43.541 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:43.541 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:43.542 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:43.543 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:43.544 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:43.545 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:43.546 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:43.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:43.547 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:43.548 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:48.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:48.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:48.549 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:48.650 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:48.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:48.652 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:48.653 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:48.654 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:48.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:48.655 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:48.656 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:53.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:53.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:53.657 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:53.708 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:53.708 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:53.758 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:53.758 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:53.758 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:53.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:53.760 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:53.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:53.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:53.763 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:53.764 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:03:58.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:58.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:03:58.766 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:03:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:03:58.817 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:58.867 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:03:58.868 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:58.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:03:58.870 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:03:58.871 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:03:58.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:03.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:03.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:03.874 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:03.925 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:03.925 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:03.977 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:03.977 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:03.977 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:03.977 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:03.978 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:03.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:03.980 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:03.982 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:03.983 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:08.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:08.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:08.985 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:09.036 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:09.036 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:09.086 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:09.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:09.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:09.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:09.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:09.091 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:09.092 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:09.093 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:14.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:14.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:14.095 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:14.200 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:14.200 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:14.200 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:14.200 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:14.200 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:14.200 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:14.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:14.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:14.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:14.203 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:14.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:14.205 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:14.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:19.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:19.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:19.207 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:19.260 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:19.260 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:19.312 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:19.312 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:19.312 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:19.312 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:19.312 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:19.312 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:19.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:19.315 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:19.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:19.317 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:19.318 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:24.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:24.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:24.320 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:24.370 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:24.421 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:24.421 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:24.421 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:24.422 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:24.422 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:24.422 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:24.422 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:24.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:24.422 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:24.423 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:24.424 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:24.425 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:24.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:24.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:24.428 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:24.430 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:29.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:29.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:29.431 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:29.482 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:29.537 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:29.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:29.540 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:29.541 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:29.542 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:29.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:29.542 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:29.543 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:34.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:34.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:34.544 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:34.595 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:34.596 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:34.647 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:34.652 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:34.653 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:39.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:39.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:39.655 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:39.744 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:39.745 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:39.746 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:39.748 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:39.750 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:44.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:44.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:44.752 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:44.802 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:44.802 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:44.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:44.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:44.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:44.857 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:44.858 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:44.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:44.858 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:44.859 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:49.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:49.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:49.860 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:49.911 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:49.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:49.964 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:49.965 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:49.966 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:49.967 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:49.968 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:49.969 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:04:54.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:54.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:04:54.971 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:04:55.022 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:04:55.022 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:04:55.075 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:04:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:04:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:04:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:04:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:04:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:04:55.075 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:55.076 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:55.077 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:04:55.078 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:04:55.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:04:55.080 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:04:55.081 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:00.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:00.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:00.083 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:00.133 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:00.134 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:00.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:00.186 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:00.187 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:00.188 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:00.189 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:00.190 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:05.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:05.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:05.191 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:05.237 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:05.237 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:05.285 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:05.285 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:05.285 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:05.285 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:05.285 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:05.285 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:05.285 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:05.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:05.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:05.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:05.289 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:05.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:05.290 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:05.291 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:10.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:10.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:10.292 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:10.344 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:10.345 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:10.397 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:10.397 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:10.397 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:10.397 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:10.397 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:10.398 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:10.399 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:10.400 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:10.400 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:10.400 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:10.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:10.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:10.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:10.400 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:10.401 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:10.401 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:10.401 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:10.401 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:10.401 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:10.401 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:10.401 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:10.402 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:10.403 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:10.405 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:15.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:15.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:15.406 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:15.457 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:15.509 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:15.509 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:15.510 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:15.511 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:15.512 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:15.513 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:15.514 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:15.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:20.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:20.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:20.517 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:20.567 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:20.567 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:20.618 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:20.618 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:20.619 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:20.620 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:20.621 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:20.622 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:20.623 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:20.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:20.624 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:20.625 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:25.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:25.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:25.626 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:25.675 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:25.726 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:25.726 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:25.726 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:25.726 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:25.726 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:25.727 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:25.728 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:25.729 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:25.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:25.731 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:25.732 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:30.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:30.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:30.734 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:30.781 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:30.782 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:30.830 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:30.830 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:30.830 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:30.830 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:30.831 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:30.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:30.833 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:30.835 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:30.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:30.835 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:30.836 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:35.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:35.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:35.837 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:35.888 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:35.941 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:35.946 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:35.947 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:40.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:40.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:40.948 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:41.048 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:41.048 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:41.050 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:41.051 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:41.052 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:41.053 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:41.054 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:46.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:46.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:46.055 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:46.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:46.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:46.156 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:46.161 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:46.162 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:51.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:51.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:51.164 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:51.209 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:51.209 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:51.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:51.261 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:51.261 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:51.261 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:51.261 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:51.261 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:51.262 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:51.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:51.265 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:05:56.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:56.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:05:56.266 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:05:56.317 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:05:56.317 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:05:56.371 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:05:56.371 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:05:56.371 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:05:56.371 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:56.372 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:56.373 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:56.374 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:56.374 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:56.374 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:56.374 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:56.374 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:05:56.375 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:05:56.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:05:56.377 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:05:56.378 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:01.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:01.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:01.379 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:01.496 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:01.496 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:01.496 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:01.497 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:01.498 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:01.499 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:01.500 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:01.501 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:01.502 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:06.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:06.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:06.504 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:06.554 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:06.555 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:06.604 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:06.604 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:06.609 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:06.610 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:11.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:11.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:11.612 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:11.680 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:11.681 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:11.744 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:11.744 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:11.744 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:11.744 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:11.744 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:11.744 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:11.744 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:11.745 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:11.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:11.747 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:11.748 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:11.749 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:11.749 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:11.749 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:11.750 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:16.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:16.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:16.751 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:16.799 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:16.799 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:16.849 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:16.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:16.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:16.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:16.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:16.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:16.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:16.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:16.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:16.854 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:16.854 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:16.854 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:16.855 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:16.856 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:16.856 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:16.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:16.856 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:16.857 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:21.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:21.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:21.859 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:21.909 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:21.962 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:21.962 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:21.962 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:21.962 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:21.964 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:21.965 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:21.966 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:21.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:21.968 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:21.968 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:21.968 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:21.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:21.968 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:21.969 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:26.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:26.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:26.970 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:27.021 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:27.021 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:27.075 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:27.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:27.077 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:27.078 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:27.079 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:27.080 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:27.081 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:32.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:32.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:32.082 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:32.176 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:32.177 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:32.180 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:32.181 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:37.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:37.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:37.182 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:37.234 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:37.234 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:37.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:37.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:37.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:37.289 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:37.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:37.291 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:37.292 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:42.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:42.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:42.293 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:42.338 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:42.383 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:42.386 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:42.387 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:42.388 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:42.388 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:42.388 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:47.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:47.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:47.390 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:47.435 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:47.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:47.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:47.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:47.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:47.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:47.489 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:47.490 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:52.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:52.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:52.491 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:52.542 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:52.542 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:52.593 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:52.593 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:52.593 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:52.594 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:52.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:52.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:52.597 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:52.598 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:52.598 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:52.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:52.598 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:52.599 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:06:57.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:57.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:06:57.600 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:06:57.648 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:06:57.648 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:06:57.696 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:06:57.696 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:06:57.696 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:06:57.696 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:06:57.696 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:06:57.696 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:06:57.696 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:57.697 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:57.698 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:06:57.699 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:06:57.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:06:57.701 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:06:57.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:02.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:02.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:02.704 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:02.754 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:02.806 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:02.811 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:02.812 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:07.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:07.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:07.814 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:07.864 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:07.864 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:07.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:07.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:07.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:07.927 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:07.928 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:07.928 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:07.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:07.928 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:07.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:12.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:12.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:12.930 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:12.980 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:13.032 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:13.032 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:13.032 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:13.032 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:13.032 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:13.032 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:13.033 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:13.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:13.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:13.037 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:13.037 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:13.037 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:13.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:13.037 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:13.038 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:18.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:18.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:18.039 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:18.090 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:18.091 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:18.143 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:18.144 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:18.145 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:18.146 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:18.147 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:18.148 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:18.149 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:23.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:23.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:23.151 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:23.202 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:23.202 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:23.257 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:23.257 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:23.257 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:23.257 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:23.257 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:23.257 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:23.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:23.260 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:23.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:23.262 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:23.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:23.262 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:23.263 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:28.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:28.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:28.264 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:28.355 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:28.358 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:28.359 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:28.360 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:28.361 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:33.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:33.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:33.363 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:33.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:33.459 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:33.459 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:33.460 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:33.461 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:33.462 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:33.463 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:33.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:33.465 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:33.466 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:38.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:38.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:38.467 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:38.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:43.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:43.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:43.571 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:43.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:43.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:43.672 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:43.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:43.674 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:43.675 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:43.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:43.675 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:43.676 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:48.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:48.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:48.677 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:48.727 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:48.727 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:48.778 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:48.778 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:48.778 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:48.778 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:48.778 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:48.778 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:48.778 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:48.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:48.780 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:48.781 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:48.782 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:48.783 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:53.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:53.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:53.784 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:53.830 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:53.830 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:53.881 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:53.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:53.882 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:53.883 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:07:58.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:58.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:07:58.884 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:07:58.935 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:07:58.935 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:07:58.987 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:07:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:07:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:07:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:07:58.987 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:07:58.988 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:58.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:07:58.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:07:58.991 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:07:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:07:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:07:58.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:07:58.992 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:07:58.993 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:03.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:03.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:03.995 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:04.045 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:04.045 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:04.098 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:04.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:04.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:04.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:04.102 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:04.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:09.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:09.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:09.105 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:09.155 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:09.156 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:09.206 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:09.206 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:09.207 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:09.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:09.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:09.210 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:09.211 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:09.212 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:14.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:14.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:14.214 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:14.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:14.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:14.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:14.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:14.322 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:14.323 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:14.323 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:14.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:14.323 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:14.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:19.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:19.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:19.325 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:19.374 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:19.423 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:19.423 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:19.424 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:19.425 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:19.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:19.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:19.428 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:19.429 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724277797.768676_101.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:24.432 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... FAIL 2024-08-21T22:08:24.432 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-7 2024-08-21T22:08:24.463 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-8 2024-08-21T22:08:24.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:24.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:24.672 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278104.490067_109.1", 2024-08-21T22:08:24.797 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:08:24.490067Z" 2024-08-21T22:08:24.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:24.798 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:08:24.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:24.798 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:08:24.799 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:08:24.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:24.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:24.799 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:24.843 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:24.843 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:24.889 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:24.889 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:24.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:24.891 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:24.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:24.893 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:24.894 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:24.895 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:29.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:29.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:29.897 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:29.947 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:29.948 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:30.005 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:35.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:35.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:35.006 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:35.100 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:35.106 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:35.107 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:40.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:40.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:40.109 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:40.163 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:40.163 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:40.218 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:40.219 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:40.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:40.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:40.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:40.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:40.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:40.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:40.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:40.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:40.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:40.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:40.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:40.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:45.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:45.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:45.227 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:45.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:45.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:45.333 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:45.334 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:45.335 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:45.336 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:45.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:45.336 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:45.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:50.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:50.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:50.338 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:50.389 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:50.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:50.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:50.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:50.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:50.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:50.447 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:50.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:50.447 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:50.448 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:08:55.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:55.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:08:55.449 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:08:55.501 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:08:55.502 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:55.557 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:55.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:55.559 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:08:55.560 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:08:55.561 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:08:55.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:08:55.562 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:08:55.563 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:00.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:00.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:00.564 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:00.615 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:00.669 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:00.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:00.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:00.672 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:00.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:00.674 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:00.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:00.674 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:00.674 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:00.674 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:00.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:00.674 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:00.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:05.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:05.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:05.677 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:05.781 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:05.781 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:05.781 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:05.782 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:05.783 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:05.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:05.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:05.786 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:05.786 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:05.786 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:05.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:05.786 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:05.787 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:10.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:10.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:10.789 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:10.835 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:10.885 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:10.885 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:10.885 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:10.885 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:10.885 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:10.885 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:10.886 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:10.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:10.886 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:10.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:10.886 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:10.886 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:10.886 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:10.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:10.888 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:10.889 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:10.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:10.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:10.892 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:10.894 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:15.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:15.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:15.896 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:15.946 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:15.946 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:15.995 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:15.996 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:15.997 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:15.998 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:15.999 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:16.000 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:16.001 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:21.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:21.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:21.003 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:21.053 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:21.054 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:21.108 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:21.108 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:21.108 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:21.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:21.110 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:21.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:21.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:21.113 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:21.114 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:26.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:26.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:26.116 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:26.219 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:26.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:26.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:26.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:26.224 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:26.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:26.224 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:26.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:26.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:26.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:31.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:31.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:31.226 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:31.279 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:31.279 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:31.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:31.335 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:31.336 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:31.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:31.339 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:31.339 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:36.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:36.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:36.341 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:36.387 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:36.434 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:36.435 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:36.436 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:36.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:36.438 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:36.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:36.439 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:36.439 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:36.439 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:36.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:36.439 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:36.440 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:41.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:41.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:41.441 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:41.492 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:41.492 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:41.546 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:41.546 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:41.546 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:41.546 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:41.546 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:41.546 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:41.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:41.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:41.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:41.550 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:41.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:41.551 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:41.552 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:46.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:46.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:46.553 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:46.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:46.658 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:46.659 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:46.660 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:46.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:46.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:46.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:46.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:46.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:46.663 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:46.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:46.663 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:46.664 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:51.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:51.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:51.666 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:51.769 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:51.769 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:51.769 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:51.769 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:51.769 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:51.769 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:51.769 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:51.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:51.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:51.772 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:51.773 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:51.774 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:51.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:51.774 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:51.775 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:09:56.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:56.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:09:56.776 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:09:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:09:56.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:09:56.880 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:09:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:09:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:09:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:09:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:09:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:09:56.880 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:56.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:56.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:09:56.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:09:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:09:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:09:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:09:56.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:09:56.885 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:09:56.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:01.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:01.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:01.887 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:01.939 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:01.939 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:01.996 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:01.996 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:06.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:06.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:06.998 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:07.046 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:07.046 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:07.096 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:07.096 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:07.097 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:07.098 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:07.099 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:07.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:07.101 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:07.102 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:12.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:12.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:12.103 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:12.154 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:12.207 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:12.208 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:12.209 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:12.210 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:12.211 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:12.212 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:12.213 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:17.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:17.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:17.214 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:17.266 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:17.266 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:17.320 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:17.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:17.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:17.324 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:17.325 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:17.325 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:17.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:17.325 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:17.326 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:22.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:22.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:22.327 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:22.372 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:22.372 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:22.422 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:22.422 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:22.422 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:22.422 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:22.423 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:22.424 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:22.425 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:22.427 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:22.427 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:27.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:27.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:27.429 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:27.481 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:27.482 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:27.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:27.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:27.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:27.548 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:27.549 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:27.550 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:32.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:32.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:32.552 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:32.603 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:32.603 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:32.655 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:32.655 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:32.655 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:32.656 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:32.658 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:32.659 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:32.660 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:32.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:37.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:37.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:37.663 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:37.708 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:37.708 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:37.759 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:37.760 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:37.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:37.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:37.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:37.764 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:37.765 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:42.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:42.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:42.767 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:42.873 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:42.873 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:42.873 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:42.873 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:42.873 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:42.874 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:42.875 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:42.876 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:42.877 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:42.878 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:42.879 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:47.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:47.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:47.881 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:47.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:47.976 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:47.977 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:47.978 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:47.979 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:47.980 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:52.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:52.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:52.981 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:53.032 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:53.032 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:53.085 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:53.085 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:53.085 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:53.085 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:53.086 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:53.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:53.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:53.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:53.090 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:53.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:53.090 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:53.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:53.090 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:53.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:10:58.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:58.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:10:58.092 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:10:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:10:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:58.195 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:10:58.196 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:58.197 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:10:58.198 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:10:58.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:10:58.200 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:10:58.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:03.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:03.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:03.201 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:03.254 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:03.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:03.310 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:03.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:03.312 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:03.313 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:03.314 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:03.315 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:08.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:08.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:08.316 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:08.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:08.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:08.419 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:08.420 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:08.421 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:08.422 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:08.423 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:08.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:08.424 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:08.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:13.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:13.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:13.427 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:13.478 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:13.479 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:13.533 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:13.534 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:13.535 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:13.536 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:13.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:13.539 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:13.539 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:18.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:18.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:18.541 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:18.591 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:18.643 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:18.643 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:18.643 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:18.643 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:18.643 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:18.644 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:18.645 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:18.646 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:18.647 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:18.648 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:18.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:23.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:23.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:23.651 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:23.703 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:23.754 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:23.755 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:23.756 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:23.757 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:23.758 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:23.759 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:23.760 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:28.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:28.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:28.762 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:28.866 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:28.866 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:28.866 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:28.866 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:28.866 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:28.866 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:28.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:28.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:28.870 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:28.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:28.871 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:28.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:33.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:33.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:33.873 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:33.920 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:33.968 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:33.969 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:33.970 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:33.971 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:33.972 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:33.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:33.974 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:33.974 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:38.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:38.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:38.976 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:39.080 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:39.080 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:39.081 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:39.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:39.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:39.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:39.085 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:39.086 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:44.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:44.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:44.088 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:44.142 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:44.142 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:44.193 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:44.194 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:44.194 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:44.194 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:44.194 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:44.194 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:44.195 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:44.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:44.195 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:44.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:44.195 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:44.196 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:44.196 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:44.196 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:44.196 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:44.196 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:44.196 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:44.197 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:44.197 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:44.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:44.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:44.197 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:44.198 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:44.198 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:44.198 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:44.198 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:44.198 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:44.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:44.199 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:44.199 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:44.199 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:44.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:44.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:44.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:44.200 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:44.200 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:44.200 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:44.200 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:44.200 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:44.200 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:44.201 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:44.202 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:44.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:44.203 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:44.203 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:44.203 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:44.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:44.204 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:44.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:49.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:49.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:49.207 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:49.258 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:49.259 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:49.311 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:49.311 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:49.311 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:49.311 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:49.311 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:49.311 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:49.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:49.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:49.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:49.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:49.315 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:49.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:49.316 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:49.317 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:54.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:54.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:54.318 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:54.366 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:54.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:54.415 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:54.416 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:54.417 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:54.418 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:54.419 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:54.420 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:54.421 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:11:59.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:59.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:11:59.423 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:11:59.475 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:11:59.476 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:11:59.530 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:11:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:11:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:11:59.534 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:11:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:11:59.536 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:11:59.536 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:04.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:04.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:04.538 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:04.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:04.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:04.645 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:04.646 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:04.647 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:04.648 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:09.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:09.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:09.650 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:09.700 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:09.700 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:09.751 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:09.751 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:09.751 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:09.751 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:09.752 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:09.753 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:09.754 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:09.755 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:09.756 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:09.757 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:14.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:14.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:14.759 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:14.862 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:14.863 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:14.864 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:14.865 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:14.866 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:14.867 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:14.868 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:19.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:19.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:19.870 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:19.973 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:19.973 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:19.974 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:19.975 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:19.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:19.977 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:19.978 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:19.979 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:24.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:24.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:24.981 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:25.031 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:25.031 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:25.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:25.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:25.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:25.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:25.090 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:25.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:25.091 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:25.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:25.091 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:25.092 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:30.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:30.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:30.093 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:30.200 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:30.202 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:30.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:30.204 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:30.204 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:30.204 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:30.204 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:30.204 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:30.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:30.205 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:30.207 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:35.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:35.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:35.208 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:35.259 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:35.313 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:35.319 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:35.320 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:40.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:40.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:40.322 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:40.374 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:40.375 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:40.486 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:40.486 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:40.486 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:40.486 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:40.486 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:40.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:40.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:40.490 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:40.491 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:40.492 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:45.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:45.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:45.494 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:45.546 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:45.547 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:45.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:45.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:45.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:45.602 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:45.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:45.604 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:45.604 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:45.604 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:45.605 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:50.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:50.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:50.606 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:50.655 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:50.708 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:50.709 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:50.710 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:50.711 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:50.712 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:50.713 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:50.713 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:50.713 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:50.714 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:12:55.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:55.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:12:55.715 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:12:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:12:55.768 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:12:55.819 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:12:55.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:12:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:12:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:12:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:12:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:12:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:12:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:12:55.825 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:12:55.826 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:00.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:00.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:00.828 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:00.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:00.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:00.932 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:00.933 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:00.934 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:00.935 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:00.936 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:00.937 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:00.938 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:05.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:05.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:05.940 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:05.985 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:06.034 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:06.034 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:06.034 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:06.034 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:06.035 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:06.037 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:06.037 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:06.038 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:06.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:06.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:06.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:06.042 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:06.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:06.042 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:06.042 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:06.042 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:06.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:06.042 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:06.043 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:11.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:11.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:11.044 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:11.149 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:11.149 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:11.150 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:11.151 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:11.152 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:11.153 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:11.154 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:11.155 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:16.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:16.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:16.157 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:16.262 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:16.263 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:16.264 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:16.265 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:16.266 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:16.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:16.267 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:16.268 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:21.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:21.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:21.269 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:21.322 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:21.377 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:21.378 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:21.379 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:21.380 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:21.381 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:21.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:21.383 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:21.384 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:26.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:26.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:26.385 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:26.436 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:26.436 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:26.490 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:26.496 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:26.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278104.490067_109.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:31.498 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... FAIL 2024-08-21T22:13:31.499 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-9 2024-08-21T22:13:31.517 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-10 2024-08-21T22:13:31.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:31.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:31.563 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:31.607 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:31.607 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:31.643 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.655 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:13:31.656 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:13:31.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.656 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:13:31.656 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.534385Z" 2024-08-21T22:13:31.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:31.656 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:13:31.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:13:31.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:31.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:31.657 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:31.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:31.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:31.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.528483_70.1", 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.528483Z", 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:31.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:31.765 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:31.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724278411.534385_73.1 target=1_1724278411.528483_70.1 2024-08-21T22:13:36.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:36.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:36.767 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:36.815 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:36.816 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:36.868 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:36.868 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:36.868 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:36.868 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:36.868 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:36.868 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:36.869 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:36.870 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:36.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:36.872 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:13:36.873 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:36.873 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:36.873 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:36.873 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:13:36.873 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:36.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:36.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:36.874 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:36.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:36.874 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:36.874 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:36.874 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:36.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:36.875 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:36.876 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:41.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:41.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:41.878 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:41.926 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:41.926 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:41.977 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:41.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:46.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:46.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:46.979 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:47.030 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:47.031 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:47.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:47.085 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:47.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:47.089 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:47.090 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:52.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:52.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:52.091 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:52.136 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:52.136 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:52.182 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:52.183 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:52.184 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:52.185 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:52.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:52.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:52.188 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:52.188 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:52.188 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:52.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:52.188 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:52.189 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:13:57.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:57.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:13:57.190 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:13:57.235 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:13:57.235 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:13:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:13:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:13:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:13:57.343 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:13:57.343 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:13:57.343 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:13:57.344 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:02.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:02.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:02.345 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:02.450 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:02.456 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:02.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:07.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:07.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:07.459 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:07.511 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:07.511 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:07.566 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:07.567 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:07.568 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:07.569 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:07.570 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:07.571 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:07.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:07.572 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:07.573 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:12.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:12.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:12.574 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:12.675 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:12.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:12.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:12.679 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:12.680 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:17.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:17.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:17.681 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:17.733 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:17.733 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:17.787 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:17.788 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:17.789 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:17.790 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:17.792 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:17.794 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:22.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:22.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:22.796 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:22.844 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:22.893 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:22.894 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:22.895 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:22.896 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:22.897 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:22.898 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:22.900 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:27.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:27.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:27.901 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:27.952 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:27.952 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:28.005 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:28.005 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:28.005 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:28.005 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:28.005 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:28.005 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:28.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:28.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:28.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:28.009 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:28.010 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:28.010 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:28.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:28.010 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:28.011 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:33.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:33.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:33.012 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:33.058 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:33.058 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:33.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:33.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:33.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:33.109 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:33.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:33.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:33.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:33.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:33.110 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:33.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:33.112 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:33.113 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:38.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:38.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:38.115 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:38.164 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:38.164 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:38.216 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:38.216 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:38.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:38.218 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:38.219 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:38.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:38.221 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:38.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:43.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:43.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:43.224 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:43.270 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:43.321 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:43.321 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:43.321 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:43.321 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:43.321 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:43.321 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:43.322 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:43.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:43.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:43.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:43.326 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:43.326 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:43.326 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:43.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:43.326 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:43.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:48.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:48.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:48.328 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:48.378 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:48.378 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:48.430 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:48.430 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:48.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:48.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:48.433 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:48.434 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:48.435 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:48.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:48.436 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:48.436 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:48.436 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:48.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:48.436 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:48.437 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:53.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:53.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:53.439 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:53.532 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:53.532 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:53.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:53.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:53.535 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:53.536 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:53.537 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:53.538 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:14:58.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:58.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:14:58.539 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:14:58.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:14:58.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:14:58.639 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:14:58.639 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:14:58.640 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:58.641 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:58.642 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:58.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:58.644 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:14:58.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:14:58.644 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:14:58.644 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:14:58.644 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:14:58.644 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:14:58.644 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:14:58.645 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:03.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:03.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:03.646 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:03.711 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:03.711 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:03.766 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:03.767 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:03.768 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:03.769 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:03.770 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:03.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:03.771 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:03.772 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:08.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:08.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:08.773 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:08.824 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:08.824 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:08.878 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:08.878 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:08.878 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:08.879 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:08.880 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:08.881 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:08.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:08.883 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:08.884 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:13.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:13.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:13.885 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:13.936 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:13.937 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:13.991 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:13.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:13.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:13.994 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:13.995 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:13.996 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:13.996 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:13.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:13.996 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:13.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:18.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:18.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:18.998 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:19.047 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:19.047 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:19.097 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:19.097 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:19.097 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:19.097 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:19.097 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:19.097 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:19.097 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:19.098 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:19.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:19.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:19.101 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:19.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:19.102 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:19.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:19.102 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:19.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:19.102 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:19.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:24.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:24.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:24.104 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:24.152 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:24.152 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:24.202 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:24.202 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:24.202 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:24.203 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:24.204 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:24.205 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:24.206 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:24.207 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:24.208 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:29.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:29.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:29.210 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:29.305 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:29.305 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:29.305 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:29.306 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:29.307 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:29.308 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:29.309 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:29.310 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:29.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:29.311 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:29.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:29.311 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:29.312 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:34.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:34.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:34.314 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:34.366 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:34.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:34.423 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:34.429 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:34.430 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:39.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:39.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:39.431 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:39.480 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:39.481 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:39.537 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:39.537 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:39.537 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:39.538 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:39.539 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:39.540 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:39.540 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:39.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:39.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:39.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:39.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:39.540 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:39.541 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:39.542 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:39.543 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:39.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:39.543 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:39.544 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:44.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:44.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:44.545 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:44.637 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:44.638 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:44.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:44.640 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:44.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:44.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:44.643 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:44.643 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:44.643 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:44.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:44.643 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:44.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:49.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:49.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:49.646 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:49.708 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:49.709 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:49.766 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:49.766 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:49.766 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:49.766 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:49.766 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:49.767 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:49.768 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:49.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:49.770 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:49.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:49.771 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:49.772 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:54.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:54.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:54.774 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:54.826 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:54.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:54.880 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:54.885 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:54.886 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:15:59.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:59.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:15:59.887 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:15:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:15:59.938 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:59.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:59.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:15:59.996 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:15:59.997 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:15:59.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:15:59.998 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:15:59.999 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:05.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:05.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:05.001 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:05.106 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:05.113 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:05.114 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:10.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:10.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:10.116 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:10.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:10.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:10.220 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:10.225 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:10.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:15.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:15.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:15.228 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:15.330 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:15.331 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:15.334 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:15.335 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:20.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:20.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:20.337 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:20.382 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:20.382 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:20.429 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:20.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:20.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:20.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:20.434 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:20.435 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:25.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:25.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:25.437 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:25.490 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:25.490 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:25.543 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:25.543 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:25.543 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:25.543 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:25.544 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:25.545 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:25.546 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:25.547 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:25.548 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:25.549 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:30.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:30.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:30.551 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:30.597 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:30.597 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:30.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:30.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:30.654 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:30.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:35.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:35.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:35.656 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:35.754 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:35.760 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:35.761 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:40.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:40.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:40.763 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:40.813 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:40.814 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:40.886 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:40.886 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:40.886 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:40.886 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:40.886 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:40.887 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:40.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:40.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:40.890 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:40.891 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:40.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:45.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:45.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:45.894 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:45.945 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:46.000 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:46.000 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:46.000 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:46.000 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:46.000 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:46.001 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:46.002 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:46.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:46.004 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:46.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:46.005 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:46.005 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:46.005 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:46.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:46.005 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:46.006 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:51.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:51.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:51.007 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:51.059 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:51.059 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:51.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:51.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:51.116 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:51.117 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:51.118 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:51.119 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:51.119 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:51.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:51.119 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:51.120 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:16:56.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:56.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:16:56.122 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:16:56.174 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:16:56.174 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:16:56.230 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:56.231 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:56.232 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:16:56.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:16:56.234 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:16:56.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:16:56.236 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:16:56.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:16:56.236 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:16:56.237 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:01.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:01.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:01.239 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:01.292 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:01.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:01.349 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:01.350 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:01.351 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:01.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:01.352 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:01.353 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:06.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:06.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:06.355 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:06.413 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:06.414 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:06.463 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:06.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:06.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:06.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:06.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:06.468 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:06.468 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:06.468 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:06.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:06.468 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:06.469 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:11.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:11.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:11.470 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:11.578 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:11.578 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:11.578 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:11.578 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:11.579 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:11.580 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:11.581 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:11.582 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:11.583 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:11.584 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:16.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:16.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:16.586 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:16.656 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:16.718 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:16.718 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:16.718 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:16.718 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:16.718 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:16.718 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:16.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:16.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:16.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:16.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:16.723 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:16.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:16.723 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:16.723 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:16.723 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:16.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:16.723 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:16.724 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:21.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:21.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:21.725 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:21.776 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:21.776 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:21.830 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:21.831 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:21.832 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:21.833 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:21.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:21.835 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:21.836 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:26.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:26.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:26.837 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:26.998 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:26.998 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:26.998 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:26.999 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:27.000 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:27.002 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:27.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:27.004 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:27.004 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:32.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:32.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:32.006 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:32.050 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:32.050 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:32.098 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:32.099 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:32.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:32.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:32.102 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:32.103 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:32.104 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:37.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:37.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:37.105 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:37.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:37.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:37.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:37.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:37.204 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:37.205 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:42.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:42.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:42.206 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:42.264 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:42.380 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:42.380 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:42.380 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:42.380 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:42.380 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:42.380 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:42.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:42.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:42.383 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:42.384 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:42.385 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:42.386 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:47.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:47.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:47.388 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:47.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:47.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:47.494 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:47.494 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:47.495 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:47.496 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:47.497 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:47.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:47.499 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:47.500 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:52.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:52.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:52.501 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:52.552 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:52.552 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:52.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:52.607 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:52.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:52.609 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:52.610 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:52.611 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:52.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:52.611 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:52.612 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:17:57.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:57.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:17:57.613 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:17:57.664 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:17:57.664 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:17:57.719 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:57.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:57.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:57.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:17:57.723 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:17:57.724 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:17:57.725 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:02.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:02.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:02.726 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:02.774 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:02.775 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:02.826 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:02.831 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:02.832 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:07.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:07.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:07.834 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:07.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:07.880 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:07.931 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:07.932 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:07.933 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:18:07.934 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:07.935 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:07.936 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:12.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:12.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:12.938 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:12.989 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:13.045 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:13.046 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:13.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:13.049 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:13.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:13.051 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:13.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:18.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:18.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:18.055 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:18.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:18.106 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:18.160 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:18.160 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:18.160 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:18.160 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:18.160 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:18.161 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:18.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:18.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:18.164 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:18.165 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:18.165 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:18.165 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:18.165 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:18.166 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:23.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:23.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:23.167 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:23.218 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:23.275 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:23.275 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:23.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:23.277 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:23.278 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:23.279 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:18:23.280 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:23.281 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:23.282 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:28.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:28.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:28.283 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:28.387 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:28.388 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:28.389 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:28.392 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:28.393 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:33.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:33.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:33.395 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:33.545 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:33.546 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:33.547 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:33.548 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.534385_73.1", 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:13:31.534385Z", 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:33.549 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:33.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:33.550 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:33.551 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:38.553 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... FAIL 2024-08-21T22:18:38.553 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-11 2024-08-21T22:18:38.575 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-12 2024-08-21T22:18:38.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:38.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:38.624 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:38.670 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:38.727 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:18:38.738 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.740 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:18:38.740 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:18:38.620288Z" 2024-08-21T22:18:38.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:38.740 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:18:38.741 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:18:38.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:38.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:38.741 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:38.800 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:38.800 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:38.886 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:38.886 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:38.886 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:38.887 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:38.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:38.889 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:38.891 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:38.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:43.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:43.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:43.893 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:43.963 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:43.963 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:44.026 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:44.026 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:44.027 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:44.027 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:44.027 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:44.027 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:44.027 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:44.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:44.028 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:44.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:44.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:44.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:44.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:44.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:44.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:44.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:44.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:44.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:44.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:44.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:44.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:44.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:44.030 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:44.030 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:44.031 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:44.031 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:44.031 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:44.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:44.031 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:44.032 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:44.032 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:44.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:44.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:44.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:44.032 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:44.033 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:44.033 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:44.033 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:44.033 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:44.033 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:44.033 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:44.034 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:44.034 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:44.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:44.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:44.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:44.034 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:44.034 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:44.035 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:44.035 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:18:44.035 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:44.035 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:44.035 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:44.036 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:44.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:44.037 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:44.038 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:49.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:49.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:49.039 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:49.089 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:49.146 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:49.146 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:49.147 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:49.148 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:49.149 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:49.150 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:49.151 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:49.152 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:54.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:54.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:54.154 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:54.204 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:54.205 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:54.260 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:54.260 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:54.261 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:54.262 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:54.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:18:54.264 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:54.265 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:54.266 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:18:59.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:59.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:18:59.267 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:18:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:18:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:18:59.374 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:18:59.374 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:18:59.374 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:18:59.374 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:18:59.374 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:18:59.374 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:18:59.374 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:18:59.375 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:59.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:59.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:18:59.378 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:18:59.379 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:18:59.380 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:04.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:04.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:04.382 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:04.426 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:04.427 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:04.529 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:04.530 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:04.531 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:04.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:04.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:04.534 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:04.536 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:09.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:09.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:09.538 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:09.637 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:09.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:09.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:09.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:09.644 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:09.644 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:09.644 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:09.644 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:09.645 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:09.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:14.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:14.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:14.649 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:14.700 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:14.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:14.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:14.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:14.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:14.764 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:14.764 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:19.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:19.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:19.766 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:19.817 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:19.817 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:19.872 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:19.873 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:19.874 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:19.875 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:19.876 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:19.877 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:19.878 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:24.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:24.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:24.879 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:24.930 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:24.930 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:24.983 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:24.988 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:24.989 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:29.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:29.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:29.991 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:30.096 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:30.098 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:30.099 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:30.099 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:30.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:30.099 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:30.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:30.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:30.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:30.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:30.102 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:30.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:35.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:35.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:35.105 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:35.170 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:35.171 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:35.236 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:35.242 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:35.243 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:40.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:40.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:40.245 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:40.295 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:40.295 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:40.350 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:40.350 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:40.351 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:40.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:40.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:40.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:40.355 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:40.356 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:45.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:45.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:45.358 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:45.408 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:45.409 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:45.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:45.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:45.465 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:45.466 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:45.467 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:45.468 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:45.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:45.468 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:45.469 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:50.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:50.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:50.470 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:50.520 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:50.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:50.580 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:50.581 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:19:55.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:55.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:19:55.582 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:19:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:19:55.633 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:19:55.688 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:19:55.688 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:19:55.688 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:19:55.688 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:19:55.688 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:19:55.688 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:55.689 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:55.690 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:19:55.691 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:55.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:55.693 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:19:55.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:19:55.693 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:19:55.693 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:19:55.693 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:19:55.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:19:55.693 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:19:55.694 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:00.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:00.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:00.696 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:00.742 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:00.743 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:00.795 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:00.795 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:00.795 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:00.795 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:00.796 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:00.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:00.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:00.799 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:00.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:00.800 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:00.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:00.800 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:00.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:05.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:05.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:05.803 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:05.850 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:05.851 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:05.902 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:05.902 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:05.903 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:05.904 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:05.905 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:05.906 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:05.907 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:05.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:10.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:10.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:10.910 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:10.955 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:10.956 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:11.005 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:11.005 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:11.006 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:11.007 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:11.009 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:11.010 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:11.011 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:11.011 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:11.011 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:11.012 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:16.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:16.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:16.013 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:16.060 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:16.061 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:16.111 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:16.111 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:16.111 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:16.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:16.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:16.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:16.115 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:16.116 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:16.117 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:21.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:21.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:21.119 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:21.170 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:21.170 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:21.224 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:21.224 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:21.224 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:21.224 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:21.224 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:21.224 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:21.225 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:21.226 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:21.227 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:21.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:21.229 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:21.230 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:26.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:26.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:26.232 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:26.281 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:26.281 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:26.335 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:26.335 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:26.336 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:26.337 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:26.338 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:26.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:26.340 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:26.341 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:26.341 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:26.342 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:31.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:31.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:31.343 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:31.394 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:31.394 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:31.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:31.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:31.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:31.453 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:31.454 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:31.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:36.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:36.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:36.457 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:36.573 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:36.573 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:36.573 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:36.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:36.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:36.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:36.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:36.578 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:36.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:41.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:41.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:41.581 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:41.625 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:41.625 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:41.690 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:41.690 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:41.691 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:41.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:41.693 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:41.694 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:41.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:41.696 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:41.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:46.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:46.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:46.699 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:46.745 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:46.796 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:46.796 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:46.796 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:46.796 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:46.796 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:46.796 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:46.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:46.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:46.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:46.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:46.800 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:46.801 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:46.801 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:46.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:46.801 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:46.802 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:51.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:51.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:51.804 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:51.849 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:51.849 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:51.958 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:51.958 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:51.959 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:51.960 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:51.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:51.962 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:51.963 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:51.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:20:56.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:56.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:20:56.965 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:20:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:20:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:20:57.077 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:57.078 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:20:57.079 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:57.080 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:20:57.081 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:20:57.082 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:20:57.083 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:02.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:02.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:02.085 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:02.134 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:02.134 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:02.188 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:02.193 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:02.194 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:07.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:07.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:07.196 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:07.263 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:07.263 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:07.323 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:07.323 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:07.323 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:07.323 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:07.323 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:07.324 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:07.325 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:07.326 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:07.327 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:07.327 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:07.327 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:07.327 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:07.327 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:07.327 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:07.327 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:07.328 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:07.329 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:07.330 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:12.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:12.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:12.332 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:12.438 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:12.438 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:12.438 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:12.439 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:12.440 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:12.441 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:12.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:12.443 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:12.444 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:17.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:17.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:17.446 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:17.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:17.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:17.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:17.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:17.562 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:17.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:17.564 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:17.565 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:22.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:22.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:22.567 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:22.629 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:22.702 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:22.702 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:22.702 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:22.702 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:22.703 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:22.704 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:22.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:22.706 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:22.707 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:22.708 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:27.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:27.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:27.709 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:27.754 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:27.754 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:27.818 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:27.818 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:27.818 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:27.818 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:27.818 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:27.818 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:27.819 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:27.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:27.824 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:32.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:32.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:32.826 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:32.877 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:32.877 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:32.933 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:32.934 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:32.935 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:32.936 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:32.937 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:32.938 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:32.940 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:37.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:37.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:37.941 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:37.993 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:37.993 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:38.049 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:38.049 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:38.051 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:38.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:38.053 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:38.054 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:38.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:43.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:43.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:43.057 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:43.108 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:43.108 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:43.163 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:43.164 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:43.165 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:43.166 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:43.167 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:43.168 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:43.170 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:48.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:48.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:48.172 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:48.234 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:48.234 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:48.286 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:48.286 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:48.286 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:48.286 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:48.286 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:48.286 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:48.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:48.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:48.289 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:48.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:48.291 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:48.291 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:48.291 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:48.291 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:48.291 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:48.292 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:53.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:53.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:53.294 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:53.342 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:53.342 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:53.399 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:53.400 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:53.401 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:53.402 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:53.403 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:53.404 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:53.405 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:53.405 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:53.405 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:53.405 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:53.406 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:21:58.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:58.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:21:58.407 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:21:58.458 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:21:58.458 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:21:58.531 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:21:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:21:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:21:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:21:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:21:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:21:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:21:58.534 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:21:58.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:21:58.536 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:21:58.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:03.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:03.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:03.539 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:03.649 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:03.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:03.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:03.652 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:03.653 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:03.654 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:03.654 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:03.655 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:08.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:08.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:08.656 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:08.708 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:08.708 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:08.773 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:08.774 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:08.775 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:08.776 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:08.777 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:08.778 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:13.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:13.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:13.780 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:13.825 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:13.825 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:13.882 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:13.882 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:13.883 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:13.884 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:13.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:13.886 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:13.887 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:13.887 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:13.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:13.887 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:13.888 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:18.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:18.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:18.889 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:18.934 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:18.934 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:19.005 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:19.005 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:19.005 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:19.005 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:19.005 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:19.006 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:19.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:19.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:19.006 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:19.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:19.007 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:19.007 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:19.007 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:19.007 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:19.007 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:19.008 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:19.008 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:19.008 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:19.008 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:19.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:19.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:19.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:19.009 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:19.009 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:19.009 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:19.010 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:19.010 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:19.010 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:19.010 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:19.011 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:19.011 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:19.011 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:19.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:19.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:19.012 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:19.012 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:19.012 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:19.012 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:19.013 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:19.013 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:19.013 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:19.013 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:19.013 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:19.014 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:19.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:19.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:19.014 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:19.014 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:19.014 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:19.015 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:19.015 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:19.015 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:19.015 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:19.015 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:19.016 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:19.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:19.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:19.016 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:19.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:19.016 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:19.017 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:19.017 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:19.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:19.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:24.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:24.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:24.021 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:24.068 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:24.069 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:24.142 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:24.142 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:24.142 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:24.142 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:24.142 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:24.143 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:24.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:24.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:24.147 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:24.147 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:24.147 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:24.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:24.147 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:24.148 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:29.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:29.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:29.149 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:29.200 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:29.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:29.260 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:29.261 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:29.262 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:29.263 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:29.264 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:34.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:34.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:34.266 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:34.395 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:34.400 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:34.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:39.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:39.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:39.403 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:39.449 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:39.505 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:39.505 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:39.505 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:39.506 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:39.507 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:39.508 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:39.509 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:39.510 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:39.511 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:44.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:44.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:44.512 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:44.559 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:44.612 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:44.612 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:44.612 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:44.612 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:44.612 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:44.613 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:44.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:44.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:44.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:44.617 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:44.617 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:44.617 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:44.617 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:44.617 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:44.618 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:49.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:49.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:49.620 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:49.671 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:49.671 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:49.736 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:49.736 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:49.736 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:49.736 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:49.737 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:49.738 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:49.739 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:49.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:49.741 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:49.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:49.741 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:49.741 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:49.741 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:49.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:49.741 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:49.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:54.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:54.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:54.743 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:54.794 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:54.794 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:54.851 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:54.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:54.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:54.855 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:54.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:54.857 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:54.858 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:22:59.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:59.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:22:59.859 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:22:59.910 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:22:59.910 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:22:59.979 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:22:59.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:59.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:22:59.983 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:22:59.984 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:22:59.985 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:22:59.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:22:59.985 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:22:59.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:04.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:04.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:04.988 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:05.035 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:05.036 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:05.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:05.096 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:05.096 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:10.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:10.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:10.098 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:10.146 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:10.219 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:10.219 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:10.219 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:10.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:10.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:10.222 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:10.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:10.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:10.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:15.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:15.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:15.226 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:15.332 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:15.333 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:15.334 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:15.335 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:15.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:15.337 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:15.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:20.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:20.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:20.339 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:20.440 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:20.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:20.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:20.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:20.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:20.445 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:20.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:20.445 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:20.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:25.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:25.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:25.447 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:25.498 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:25.498 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:25.573 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:25.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:25.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:25.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:25.577 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:25.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:25.578 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:25.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:30.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:30.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:30.580 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:30.690 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:30.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:30.692 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:30.693 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:30.694 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:30.695 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:30.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:30.695 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:30.696 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:35.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:35.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:35.697 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:35.800 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:35.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:40.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:40.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:40.802 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:40.854 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:40.936 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:40.936 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:40.936 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:40.936 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:40.936 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:40.937 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:40.937 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:40.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:40.937 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:40.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:40.937 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:40.938 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:40.939 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:40.940 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:40.941 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:40.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:40.942 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:40.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:40.942 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:40.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:40.942 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:40.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:45.945 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... FAIL 2024-08-21T22:23:45.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:45.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:45.973 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:46.026 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:46.026 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:46.083 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:23:46.102 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:23:46.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.102 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:46.102 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:23:46.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:23:45.971994Z" 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:46.103 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:23:46.104 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:23:46.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:46.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:46.105 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:46.148 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:46.149 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:46.205 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:46.205 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:46.205 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:46.205 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:46.206 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:46.207 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:46.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278718.620288_92.1", 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:18:38.620288Z", 2024-08-21T22:23:46.209 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:46.210 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:46.211 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724279025.971994_109.1 target=1_1724278718.620288_92.1 2024-08-21T22:23:51.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:51.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:51.213 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:51.267 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:51.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:51.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:51.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:51.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:51.357 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:23:56.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:56.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:23:56.359 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:23:56.410 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:23:56.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:56.477 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:56.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:56.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:56.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:23:56.481 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:23:56.482 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:23:56.482 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:23:56.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:23:56.482 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:23:56.483 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:01.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:01.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:01.485 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:01.531 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:01.531 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:01.615 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:01.615 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:01.615 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:01.615 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:01.615 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:01.616 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:01.617 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:01.618 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:01.619 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:01.620 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:01.621 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:06.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:06.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:06.623 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:06.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:06.721 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:06.722 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:06.723 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:06.724 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:06.725 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:11.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:11.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:11.726 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:11.777 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:11.777 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:11.860 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:11.860 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:11.860 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:11.860 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:11.860 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:11.861 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:11.862 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:11.863 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:11.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:11.864 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:11.864 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:11.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:11.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:11.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:11.865 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:11.865 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:11.865 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:11.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:11.865 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:11.866 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:11.866 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:11.866 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:11.866 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:11.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:11.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:11.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:11.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:11.867 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:11.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:11.867 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:11.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:11.868 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:11.869 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:16.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:16.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:16.870 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:16.922 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:16.925 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:16.989 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:16.989 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:16.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:16.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:16.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:16.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:16.992 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:16.992 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:16.992 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:16.992 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:16.992 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:16.993 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:16.994 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:16.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:16.996 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:16.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:16.996 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:16.997 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:21.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:21.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:21.998 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:22.105 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:22.105 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:22.106 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:22.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:22.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:22.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:22.110 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:22.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:22.110 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:22.111 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:27.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:27.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:27.112 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:27.231 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:27.231 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:27.232 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:27.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:27.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:27.235 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:27.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:27.236 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:27.237 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:32.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:32.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:32.239 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:32.350 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:32.350 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:32.351 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:32.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:32.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:32.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:32.355 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:32.356 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:37.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:37.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:37.358 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:37.410 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:37.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:37.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:37.483 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:37.484 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:37.485 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:37.485 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:37.485 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:37.485 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:37.485 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:37.485 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:37.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:37.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:37.487 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:37.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:42.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:42.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:42.490 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:42.600 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:42.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:42.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:42.606 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:42.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:42.606 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:42.606 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:42.606 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:42.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:42.606 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:42.607 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:47.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:47.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:47.608 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:47.660 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:47.660 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:47.733 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:47.734 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:47.735 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:47.736 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:47.737 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:47.738 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:47.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:47.739 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:47.740 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:52.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:52.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:52.741 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:52.912 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:52.912 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:52.913 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:52.914 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:52.915 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:52.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:52.917 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:52.918 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:24:57.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:57.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:24:57.919 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:24:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:24:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:24:58.024 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:58.025 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:24:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:24:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:24:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:24:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:24:58.030 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:24:58.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:03.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:03.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:03.032 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:03.156 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:03.156 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:03.157 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:03.158 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:03.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:03.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:03.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:03.161 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:03.162 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:08.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:08.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:08.163 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:08.213 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:08.213 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:08.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:08.280 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:08.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:13.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:13.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:13.283 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:13.332 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:13.332 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:13.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:13.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:13.418 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:13.419 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:13.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:13.421 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:13.422 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:18.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:18.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:18.424 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:18.476 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:18.555 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:18.555 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:18.555 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:18.556 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:18.557 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:18.557 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:18.557 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:18.557 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:18.557 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:18.558 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:18.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:18.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:18.561 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:18.562 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:18.562 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:18.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:18.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:18.562 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:18.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:18.563 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:18.563 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:18.563 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:18.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:18.563 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:18.564 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:23.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:23.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:23.566 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:23.672 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:23.672 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:23.672 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:23.672 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:23.672 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:23.672 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:23.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:23.674 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:23.675 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:23.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:23.677 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:23.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:28.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:28.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:28.680 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:28.730 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:28.731 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:28.801 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:28.801 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:28.801 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:28.802 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:28.803 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:28.804 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:28.805 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:28.806 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:28.807 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:33.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:33.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:33.808 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:33.859 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:33.859 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:33.937 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:33.937 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:33.937 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:33.937 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:33.937 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:33.938 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:33.939 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:33.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:33.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:33.942 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:33.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:33.942 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:33.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:33.942 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:33.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:33.942 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:33.943 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:38.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:38.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:38.944 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:39.054 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:39.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:39.056 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:39.058 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:39.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:44.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:44.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:44.061 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:44.248 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:44.248 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:44.249 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:44.250 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:44.252 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:44.253 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:44.254 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:49.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:49.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:49.256 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:49.308 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:49.309 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:49.375 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:49.375 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:49.376 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:49.377 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:49.378 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:49.379 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:49.380 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:49.381 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:54.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:54.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:54.387 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:54.506 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:54.506 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:54.507 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:54.508 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:54.509 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:54.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:54.511 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:54.511 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:54.512 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:25:59.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:59.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:25:59.514 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:25:59.559 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:25:59.560 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:59.641 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:59.642 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:59.643 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:25:59.644 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:25:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:25:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:25:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:25:59.646 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:25:59.646 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:25:59.647 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:04.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:04.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:04.648 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:04.760 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:04.760 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:04.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:04.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:04.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:04.764 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:04.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:04.765 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:04.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:09.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:09.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:09.768 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:09.816 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:09.816 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:09.870 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:09.870 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:09.870 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:09.870 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:09.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:09.872 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:09.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:09.874 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:09.875 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:09.876 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:14.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:14.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:14.877 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:14.928 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:14.929 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:14.988 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:14.989 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:14.990 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:14.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:14.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:14.993 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:14.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:14.994 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:14.995 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:19.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:19.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:19.996 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:20.047 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:20.047 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:20.117 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:20.117 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:20.117 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:20.117 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:20.117 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:20.117 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:20.117 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:20.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:20.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:20.121 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:20.122 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:20.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:20.122 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:20.123 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:25.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:25.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:25.124 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:25.176 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:25.261 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:25.262 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:25.263 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:25.264 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:25.265 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:25.266 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:25.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:25.267 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:25.268 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:30.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:30.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:30.269 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:30.388 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:30.388 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:30.388 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:30.388 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:30.389 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:30.390 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:30.391 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:30.392 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:30.393 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:30.394 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:35.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:35.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:35.396 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:35.448 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:35.449 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:35.513 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:35.520 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:35.521 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:40.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:40.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:40.523 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:40.573 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:40.574 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:40.660 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:40.660 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:40.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:40.660 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:40.661 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:40.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:40.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:40.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:40.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:40.665 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:40.665 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:40.665 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:40.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:40.665 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:40.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:45.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:45.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:45.667 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:45.717 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:45.717 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:45.794 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:45.794 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:45.794 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:45.794 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:45.794 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:45.795 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:45.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:45.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:45.798 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:45.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:45.799 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:45.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:45.799 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:45.800 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:50.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:50.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:50.801 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:50.912 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:50.913 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:50.914 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:50.915 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:50.916 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:50.917 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:50.918 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:50.918 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:50.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:50.918 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:50.919 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:26:55.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:55.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:26:55.920 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:26:55.967 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:26:55.967 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:26:56.059 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:26:56.059 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:56.060 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:26:56.061 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:26:56.062 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:26:56.063 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:26:56.064 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:26:56.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:01.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:01.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:01.067 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:01.175 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:01.176 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:01.177 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:01.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:01.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:01.178 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:01.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:01.184 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:01.186 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:06.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:06.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:06.187 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:06.235 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:06.295 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:06.296 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:06.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:06.298 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:06.299 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:06.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:06.301 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:06.302 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:11.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:11.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:11.303 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:11.364 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:11.427 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:11.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:11.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:11.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:11.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:11.432 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:11.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:11.433 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:11.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:11.433 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:11.434 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:16.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:16.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:16.436 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:16.481 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:16.481 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:16.554 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:16.554 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:16.555 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:16.556 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:16.557 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:16.558 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:16.559 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:16.560 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:21.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:21.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:21.562 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:21.614 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:21.614 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:21.769 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:21.769 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:21.769 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:21.769 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:21.770 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:21.771 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:21.772 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:21.773 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:21.774 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:21.775 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:26.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:26.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:26.777 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:26.828 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:26.898 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:26.898 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:26.898 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:26.899 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:26.900 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:26.901 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:26.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:26.903 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:26.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:26.903 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:26.903 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:26.903 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:26.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:26.903 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:26.904 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:31.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:31.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:31.906 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:31.956 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:31.956 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:32.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:32.037 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:32.038 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:32.039 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:32.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:32.041 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:32.041 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:37.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:37.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:37.043 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:37.096 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:37.096 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:37.170 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:37.171 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:37.172 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:37.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:37.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:37.175 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:37.176 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:42.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:42.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:42.177 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:42.283 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:42.283 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:42.283 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:42.283 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:42.283 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:42.283 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:42.284 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:42.285 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:42.286 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:42.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:42.288 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:42.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:42.289 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:42.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:42.289 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:42.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:47.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:47.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:47.292 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:47.340 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:47.340 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:47.402 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:47.403 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:47.404 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:47.407 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:47.408 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:52.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:52.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:52.410 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:52.461 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:52.461 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:52.519 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:52.520 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:52.521 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:52.522 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:52.523 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:52.524 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:52.525 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:52.525 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:52.526 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:27:57.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:57.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:27:57.527 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:27:57.578 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:27:57.578 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:27:57.637 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:27:57.638 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:27:57.638 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:27:57.638 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:27:57.638 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:27:57.639 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:27:57.639 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:57.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:57.639 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:27:57.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:57.639 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:27:57.640 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:57.640 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:57.640 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:27:57.640 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:57.640 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:27:57.640 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:57.641 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:27:57.641 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:57.641 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:57.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:57.641 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:57.641 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:27:57.642 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:57.642 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:57.642 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:27:57.642 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:57.642 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:57.642 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:57.643 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:27:57.643 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:57.643 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:57.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:57.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:57.644 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:27:57.644 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:57.644 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:57.644 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:27:57.645 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:57.645 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:57.645 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:57.645 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:27:57.645 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:57.645 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:57.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:57.646 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:27:57.646 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:27:57.646 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:27:57.646 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:27:57.647 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:27:57.647 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:27:57.647 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:27:57.647 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:27:57.648 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:27:57.648 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:27:57.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:57.648 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:57.648 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:27:57.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:27:57.649 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:27:57.649 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:27:57.649 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:27:57.649 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:27:57.649 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:27:57.651 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:02.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:02.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:02.653 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:02.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:07.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:07.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:07.769 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:07.820 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:07.875 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:07.876 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:07.877 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:07.878 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:07.879 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:07.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:07.881 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:07.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:12.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:12.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:12.882 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:12.936 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:12.936 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:12.996 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:12.996 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:12.996 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:12.996 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:12.997 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:12.998 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:12.999 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:13.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:13.001 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:13.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:13.001 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:13.001 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:13.001 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:13.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:13.001 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:13.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:18.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:18.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:18.003 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:18.048 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:18.048 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:18.117 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:18.117 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:18.118 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:18.118 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:18.118 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:18.118 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:18.118 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:18.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:18.118 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:18.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:18.120 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:18.121 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:18.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:18.123 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:18.123 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:23.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:23.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:23.125 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:23.177 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:23.247 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:23.247 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:23.247 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:23.247 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:23.247 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:23.247 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:23.247 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:23.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:23.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:23.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:23.251 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:23.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:23.252 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:23.252 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:23.252 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:23.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:23.252 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:23.253 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:28.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:28.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:28.255 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:28.390 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:28.390 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:28.391 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:28.392 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:28.393 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:28.394 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:28.395 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:28.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:28.395 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:28.395 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:28.395 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:28.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:28.395 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:28.396 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:33.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:33.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:33.397 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:33.448 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:33.448 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:33.514 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:33.514 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:33.514 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:33.515 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:33.516 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:33.517 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:33.518 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:33.519 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:33.520 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:38.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:38.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:38.522 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:38.660 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:38.660 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:38.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:38.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:38.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:38.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:38.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:38.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:38.665 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:38.665 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:38.665 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:38.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:38.665 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:38.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:43.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:43.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:43.667 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:43.717 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:43.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:43.788 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:43.788 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:43.788 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:43.789 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:43.789 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:43.789 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:43.789 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:43.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:43.789 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:43.790 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:43.790 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:43.790 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:43.790 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:43.790 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:43.791 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:43.791 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:43.791 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:43.791 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:43.791 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:43.792 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:43.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:43.792 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:43.792 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:43.792 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:43.792 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:43.793 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:43.793 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:43.793 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:43.793 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:43.793 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:43.794 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:43.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:43.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:43.794 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:43.794 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:43.795 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:43.795 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:43.795 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:43.795 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:43.795 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:43.795 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:43.796 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:43.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:43.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:43.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:43.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:43.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:43.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:43.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:43.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:43.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:43.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:43.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:43.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:43.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:43.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:43.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:43.798 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:43.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:43.799 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:43.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:43.799 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:43.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:43.800 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:43.802 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:48.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:48.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:48.804 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:48.854 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:48.935 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:48.935 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:48.935 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:48.935 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:48.935 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:48.935 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:48.935 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:48.936 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:48.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:48.938 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279025.971994_109.1", 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:48.939 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:23:45.971994Z", 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:48.940 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:48.941 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:28:53.943 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... FAIL 2024-08-21T22:28:53.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:53.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:53.966 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:54.013 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:54.064 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:28:54.076 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:28:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:28:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:28:53.963733Z" 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:54.077 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:28:54.078 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:28:54.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:54.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:54.079 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:54.122 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:54.122 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.179 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:54.180 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:54.181 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.955589_115.1", 2024-08-21T22:28:54.182 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.955589Z", 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:54.183 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:54.184 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724279333.963733_118.1 target=1_1724279333.955589_115.1 2024-08-21T22:28:59.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:59.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:28:59.186 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:28:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:28:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:28:59.331 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:28:59.331 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:28:59.331 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:28:59.331 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:28:59.331 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:28:59.331 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:28:59.332 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:28:59.333 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:28:59.334 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:28:59.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:28:59.336 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:28:59.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:04.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:04.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:04.339 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:04.383 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:04.383 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:04.467 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:04.467 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:04.467 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:04.467 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:04.467 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:04.467 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:04.468 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:04.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:04.470 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:04.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:04.472 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:04.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:04.472 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:04.472 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:04.472 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:04.472 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:04.472 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:04.473 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:09.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:09.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:09.474 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:09.524 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:09.525 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:09.594 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:09.594 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:09.594 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:09.594 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:09.594 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:09.594 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:09.594 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:09.595 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:09.596 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:09.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:09.598 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:09.599 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:09.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:14.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:14.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:14.602 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:14.653 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:14.653 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:14.730 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:14.730 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:14.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:14.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:14.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:14.735 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:14.735 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:14.736 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:19.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:19.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:19.737 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:19.787 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:19.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:19.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:19.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:19.872 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:19.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:19.872 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:19.873 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:24.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:24.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:24.874 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:24.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:24.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:24.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:24.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:24.990 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:24.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:24.991 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:24.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:24.991 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:24.992 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:29.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:29.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:29.993 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:30.126 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:30.126 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:30.126 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:30.126 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:30.127 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:30.128 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:30.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:30.130 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:30.131 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:30.132 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:35.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:35.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:35.134 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:35.266 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:35.275 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:35.276 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:40.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:40.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:40.277 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:40.328 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:40.390 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:40.391 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:40.392 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:40.393 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:40.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:40.395 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:40.396 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:40.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:40.398 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:40.398 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:40.398 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:40.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:40.398 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:40.400 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:45.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:45.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:45.401 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:45.451 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:45.452 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:45.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:45.544 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:45.545 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:45.546 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:45.547 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:45.548 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:50.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:50.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:50.550 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:50.601 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:50.601 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:50.690 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:50.691 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:29:55.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:55.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:29:55.692 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:29:55.744 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:29:55.744 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:29:55.806 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:29:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:29:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:29:55.812 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:29:55.812 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:29:55.812 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:29:55.812 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:29:55.813 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:00.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:00.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:00.815 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:00.948 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:00.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:05.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:05.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:05.951 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:06.085 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:06.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:06.090 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:06.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:11.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:11.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:11.092 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:11.143 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:11.143 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:11.202 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:11.202 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:11.202 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:11.202 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:11.202 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:11.203 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:11.204 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:11.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:11.206 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:11.207 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:11.208 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:16.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:16.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:16.209 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:16.257 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:16.340 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:16.346 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:16.347 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:21.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:21.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:21.348 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:21.398 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:21.398 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:21.456 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:21.457 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:21.458 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:21.459 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:21.460 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:21.461 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:26.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:26.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:26.463 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:26.513 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:26.514 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:26.594 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:26.600 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:26.601 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:31.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:31.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:31.603 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:31.652 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:31.653 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:31.729 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:31.729 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:31.730 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:31.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:31.732 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:31.733 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:31.734 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:31.736 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:36.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:36.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:36.737 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:36.787 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:36.788 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:36.850 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:36.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:36.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:36.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:36.855 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:36.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:36.855 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:36.856 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:41.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:41.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:41.857 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:41.959 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:41.960 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:41.964 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:41.965 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:46.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:46.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:46.966 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:47.017 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:47.018 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:47.109 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:47.109 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:47.109 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:47.109 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:47.110 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:47.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:47.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:47.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:47.114 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:47.115 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:52.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:52.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:52.117 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:52.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:52.168 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:52.232 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:52.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:52.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:52.235 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:52.236 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:52.237 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:52.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:52.237 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:52.238 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:30:57.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:57.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:30:57.239 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:30:57.291 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:30:57.291 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:30:57.352 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:30:57.352 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:30:57.352 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:30:57.352 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:30:57.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:57.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:57.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:30:57.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:30:57.357 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:30:57.358 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:02.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:02.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:02.359 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:02.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:02.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:02.497 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:02.498 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:07.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:07.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:07.501 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:07.552 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:07.553 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:07.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:07.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:07.612 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:07.613 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:07.614 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:07.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:07.615 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:07.616 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:12.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:12.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:12.617 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:12.663 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:12.663 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:12.729 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:12.729 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:12.730 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:12.731 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:12.732 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:12.733 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:12.734 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:12.735 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:17.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:17.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:17.737 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:17.788 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:17.788 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:17.849 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:17.849 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:17.849 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:17.850 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:17.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:17.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:17.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:17.854 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:17.855 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:22.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:22.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:22.856 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:22.974 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:22.981 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:22.982 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:27.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:27.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:27.983 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:28.100 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:28.100 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:28.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:28.102 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:28.103 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:28.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:28.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:28.105 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:28.106 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:33.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:33.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:33.107 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:33.157 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:33.219 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:33.219 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:33.219 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:33.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:33.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:38.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:38.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:38.226 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:38.275 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:38.276 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:38.360 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:38.365 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:38.366 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:43.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:43.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:43.367 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:43.413 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:43.413 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:43.470 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:43.470 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:43.470 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:43.471 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:43.472 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:43.473 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:43.474 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:43.475 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:43.477 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:48.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:48.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:48.478 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:48.589 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:48.589 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:48.589 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:48.589 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:48.590 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:48.591 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:48.592 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:48.593 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:48.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:48.595 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:48.596 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:53.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:53.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:53.597 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:53.642 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:53.642 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:53.710 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:53.711 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:53.712 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:53.713 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:53.714 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:53.715 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:53.716 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:53.717 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:53.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:53.718 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:53.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:53.718 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:53.719 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:31:58.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:58.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:31:58.721 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:31:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:31:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:31:58.854 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:31:58.854 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:31:58.854 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:31:58.854 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:58.855 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:58.856 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:58.857 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:31:58.858 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:31:58.859 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:31:58.860 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:31:58.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:31:58.860 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:31:58.861 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:03.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:03.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:03.862 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:03.981 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:03.982 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:03.983 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:03.984 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:03.985 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:03.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:08.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:08.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:08.988 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:09.038 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:09.039 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:09.098 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:09.098 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:09.098 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:09.099 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:09.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:09.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:09.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:09.103 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:09.104 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:14.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:14.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:14.106 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:14.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:14.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:14.251 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:14.251 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:14.251 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:14.251 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:14.251 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:14.252 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:14.253 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:14.254 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:14.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:14.256 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:14.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:14.256 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:14.256 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:14.256 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:14.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:14.256 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:14.257 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:19.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:19.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:19.259 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:19.311 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:19.311 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:19.370 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:19.370 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:19.371 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:19.371 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:19.371 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:19.371 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:19.371 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:19.372 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:19.373 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:19.374 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:19.374 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:19.374 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:19.374 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:19.374 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:19.375 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:19.375 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:19.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:19.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:19.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:19.375 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:19.376 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:19.376 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:19.376 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:19.376 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:19.376 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:19.376 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:19.376 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:19.377 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:19.378 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:19.379 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:19.380 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:24.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:24.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:24.381 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:24.519 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:24.519 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:24.519 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:24.519 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:24.519 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:24.521 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:24.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:24.523 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:24.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:24.524 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:24.524 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:29.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:29.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:29.525 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:29.576 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:29.577 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:29.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:29.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:29.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:29.663 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:29.664 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:29.665 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:34.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:34.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:34.667 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:34.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:34.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:34.783 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:39.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:39.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:39.785 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:39.850 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:39.850 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:39.956 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:39.956 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:39.956 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:39.956 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:39.956 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:39.956 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:39.956 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:39.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:39.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:39.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:39.961 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:39.961 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:39.961 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:39.962 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:44.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:44.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:44.963 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:45.013 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:45.080 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:45.080 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:45.080 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:45.080 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:45.080 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:45.081 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:45.082 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:45.083 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:45.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:45.085 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:45.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:45.085 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:45.085 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:45.085 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:45.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:45.085 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:45.086 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:50.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:50.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:50.087 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:50.201 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:50.202 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:50.203 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:50.204 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:50.205 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:50.205 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:50.206 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:32:55.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:55.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:32:55.207 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:32:55.258 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:32:55.258 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:32:55.351 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:32:55.351 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:32:55.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:55.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:55.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:32:55.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:32:55.356 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:32:55.357 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:00.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:00.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:00.358 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:00.409 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:00.409 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:00.473 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:00.474 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:00.475 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:00.476 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:00.477 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:00.479 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:00.479 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:05.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:05.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:05.481 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:05.546 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:05.546 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:05.675 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:05.675 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:05.675 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:05.676 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:05.677 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:05.678 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:05.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:05.680 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:05.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:10.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:10.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:10.683 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:10.728 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:10.728 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:10.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:10.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:10.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:10.798 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:10.799 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:10.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:10.800 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:10.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:15.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:15.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:15.802 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:15.848 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:15.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:15.952 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:15.953 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:20.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:20.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:20.955 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:21.006 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:21.006 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:21.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:21.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:21.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:21.067 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:21.068 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:21.069 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:21.069 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:21.070 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:26.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:26.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:26.071 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:26.195 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:26.196 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:26.197 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:26.198 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:26.199 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:26.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:31.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:31.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:31.202 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:31.252 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:31.252 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:31.326 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:31.326 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:31.326 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:31.327 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:31.330 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:31.331 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:31.332 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:36.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:36.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:36.334 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:36.384 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:36.440 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:36.440 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:36.441 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:36.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:36.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:36.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:36.445 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:36.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:41.447 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:41.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:41.448 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:41.499 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:41.565 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:41.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:41.569 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:41.570 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:41.571 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:46.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:46.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:46.573 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:46.624 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:46.624 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:46.680 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:46.681 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:46.682 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:46.683 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:46.684 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:46.685 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:46.685 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:46.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:51.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:51.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:51.688 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:51.739 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:51.822 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:51.822 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:51.822 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:51.822 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:51.822 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:51.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:51.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:51.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:51.826 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:51.827 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:51.828 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:33:56.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:56.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:33:56.830 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:33:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:33:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:56.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:56.940 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:33:56.941 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:33:56.942 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279333.963733_118.1", 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:28:53.963733Z", 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:33:56.943 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:33:56.944 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:01.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... FAIL 2024-08-21T22:34:01.947 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-15 2024-08-21T22:34:01.978 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-16 2024-08-21T22:34:02.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:02.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:02.028 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:02.073 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:02.073 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:02.131 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:34:02.150 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:34:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:34:02.023939Z" 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:02.151 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:34:02.152 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:34:02.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:02.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:02.153 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:02.196 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:02.196 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:02.257 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:02.262 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:02.263 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:07.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:07.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:07.264 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:07.316 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:07.317 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:07.382 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:07.382 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:07.382 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:07.383 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:07.384 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:07.385 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:07.386 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:07.387 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:07.388 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:12.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:12.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:12.390 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:12.440 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:12.441 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:12.508 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:12.508 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:12.508 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:12.509 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:12.510 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:12.511 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:12.512 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:12.513 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:12.514 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:12.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:12.514 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:12.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:17.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:17.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:17.516 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:17.568 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:17.655 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:17.656 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:17.657 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:17.658 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:17.659 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:17.660 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:17.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:22.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:22.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:22.663 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:22.706 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:22.706 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:22.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:22.783 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:22.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:22.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:22.787 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:22.788 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:27.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:27.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:27.790 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:27.897 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:27.897 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:27.898 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:27.899 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:27.900 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:27.901 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:27.902 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:27.903 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:32.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:32.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:32.905 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:32.956 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:33.022 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:33.022 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:33.022 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:33.022 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:33.023 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:33.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:33.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:33.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:33.027 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:33.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:33.027 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:33.027 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:33.027 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:33.027 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:33.027 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:33.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:38.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:38.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:38.030 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:38.080 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:38.080 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:38.139 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:38.140 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:38.141 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:38.142 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:38.143 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:38.144 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:38.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:43.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:43.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:43.147 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:43.264 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:43.264 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:43.264 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:43.264 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:43.264 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:43.265 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:43.266 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:43.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:43.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:43.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:43.267 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:43.267 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:43.267 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:43.267 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:43.268 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:43.269 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:43.270 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:43.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:43.270 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:43.271 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:48.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:48.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:48.272 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:48.377 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:48.377 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:48.377 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:48.377 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:48.377 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:48.378 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:48.379 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:48.380 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:48.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:48.382 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:48.383 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:53.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:53.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:53.384 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:53.429 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:53.429 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:53.483 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:53.483 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:53.483 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:53.483 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:53.484 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:53.485 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:53.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:53.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:53.488 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:53.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:53.488 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:53.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:53.488 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:53.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:53.488 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:53.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:34:58.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:58.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:34:58.490 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:34:58.536 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:34:58.536 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:34:58.666 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:34:58.666 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:34:58.666 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:34:58.666 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:34:58.666 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:34:58.666 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:34:58.666 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:58.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:58.667 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:34:58.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:58.667 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:34:58.667 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:58.667 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:58.667 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:58.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:58.669 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:34:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:34:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:34:58.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:34:58.672 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:34:58.672 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:03.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:03.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:03.674 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:03.720 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:03.785 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:03.785 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:03.785 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:03.785 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:03.785 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:03.787 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:03.788 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:03.789 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:03.790 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:03.791 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:08.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:08.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:08.792 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:08.838 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:08.838 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:08.924 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:08.925 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:08.926 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:08.927 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:08.928 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:08.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:13.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:13.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:13.930 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:13.981 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:14.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:14.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:14.047 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:14.048 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:14.049 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:14.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:14.050 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:14.051 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:19.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:19.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:19.052 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:19.169 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:19.169 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:19.169 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:19.170 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:19.171 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:19.172 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:19.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:19.174 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:19.175 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:24.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:24.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:24.177 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:24.227 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:24.227 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:24.308 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:24.308 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:24.308 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:24.308 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:24.309 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:24.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:24.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:24.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:24.313 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:24.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:24.313 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:24.314 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:29.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:29.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:29.315 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:29.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:29.440 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:29.440 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:29.440 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:29.440 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:29.441 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:29.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:29.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:29.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:29.445 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:29.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:34.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:34.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:34.448 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:34.499 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:34.500 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:34.558 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:34.564 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:34.565 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:39.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:39.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:39.566 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:39.610 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:39.610 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:39.685 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:39.685 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:39.687 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:39.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:39.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:39.690 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:39.690 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:44.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:44.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:44.691 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:44.737 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:44.737 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:44.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:44.792 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:44.793 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:44.794 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:44.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:44.796 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:44.796 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:44.796 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:44.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:44.796 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:44.797 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:49.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:49.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:49.798 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:49.850 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:49.912 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:49.912 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:49.912 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:49.912 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:49.913 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:49.914 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:49.915 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:49.916 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:49.917 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:49.918 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:35:54.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:54.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:35:54.920 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:35:54.971 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:35:54.972 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:35:55.038 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:35:55.038 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:55.039 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:55.040 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:55.041 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:35:55.042 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:35:55.043 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:35:55.044 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:00.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:00.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:00.047 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:00.098 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:00.098 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:00.172 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:00.172 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:00.172 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:00.172 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:00.172 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:00.173 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:00.174 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:00.175 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:00.176 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:00.176 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:00.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:00.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:00.178 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:00.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:00.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:00.178 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:00.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:00.179 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:00.180 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:05.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:05.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:05.181 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:05.232 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:05.232 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:05.307 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:05.308 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:05.309 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:05.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:05.311 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:05.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:05.312 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:05.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:05.312 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:05.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:05.312 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:05.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:10.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:10.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:10.314 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:10.365 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:10.366 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:10.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:10.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:10.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:10.435 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:10.436 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:10.437 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:10.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:10.437 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:10.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:15.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:15.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:15.440 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:15.501 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:15.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:15.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:15.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:15.601 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:15.602 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:15.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:15.603 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:15.603 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:20.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:20.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:20.605 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:20.656 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:20.656 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:20.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:20.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:20.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:20.720 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:20.721 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:20.722 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:25.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:25.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:25.724 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:25.775 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:25.775 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:25.855 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:25.855 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:25.855 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:25.855 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:25.856 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:25.857 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:25.858 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:25.859 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:25.860 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:25.861 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:30.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:30.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:30.862 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:30.986 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:30.986 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:30.986 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:30.986 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:30.986 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:30.986 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:30.987 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:30.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:30.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:30.990 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:30.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:30.991 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:30.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:30.991 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:30.992 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:35.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:35.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:35.993 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:36.044 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:36.044 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:36.103 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:36.103 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:36.103 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:36.103 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:36.103 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:36.104 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:36.105 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:36.106 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:36.107 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:36.107 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:36.107 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:36.107 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:36.107 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:36.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:36.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:36.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:36.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:36.108 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:36.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:36.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:36.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:36.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:36.109 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:36.109 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:36.110 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:36.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:36.111 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:36.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:41.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:41.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:41.113 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:41.164 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:41.164 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:41.243 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:41.243 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:41.243 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:41.243 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:41.243 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:41.243 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:41.243 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:41.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:41.245 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:41.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:41.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:41.248 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:41.248 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:41.248 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:41.248 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:41.248 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:41.249 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:46.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:46.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:46.250 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:46.301 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:46.371 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:46.372 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:46.373 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:46.374 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:46.375 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:46.376 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:51.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:51.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:51.377 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:51.427 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:51.428 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:51.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:51.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:51.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:51.489 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:51.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:51.490 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:51.490 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:51.490 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:51.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:51.490 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:51.491 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:36:56.492 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:56.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:36:56.492 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:36:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:36:56.622 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:36:56.622 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:36:56.622 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:36:56.622 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:36:56.622 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:36:56.622 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:56.623 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:36:56.624 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:56.626 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:36:56.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:36:56.627 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:36:56.627 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:36:56.627 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:36:56.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:36:56.627 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:36:56.628 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:01.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:01.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:01.630 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:01.680 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:01.680 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:01.739 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:01.744 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:01.746 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:06.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:06.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:06.747 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:06.798 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:06.799 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:06.886 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:06.886 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:06.886 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:06.886 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:06.886 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:06.886 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:06.886 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:06.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:06.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:06.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:06.891 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:06.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:11.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:11.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:11.894 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:11.945 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:12.011 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:12.012 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:12.013 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:12.014 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:12.015 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:12.016 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:12.017 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:17.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:17.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:17.019 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:17.071 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:17.072 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:17.147 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:17.147 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:17.147 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:17.148 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:17.149 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:17.150 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:17.152 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:17.153 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:22.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:22.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:22.154 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:22.202 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:22.202 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:22.290 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:22.291 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:22.292 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:22.293 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:22.294 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:22.295 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:22.296 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:27.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:27.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:27.298 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:27.349 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:27.349 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:27.408 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:27.408 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:27.408 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:27.408 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:27.408 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:27.408 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:27.408 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:27.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:27.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:27.411 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:27.412 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:27.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:27.413 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:27.414 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:32.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:32.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:32.415 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:32.466 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:32.466 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:32.540 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:32.541 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:32.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:32.543 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:32.544 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:32.545 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:32.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:32.546 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:32.547 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:37.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:37.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:37.548 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:37.666 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:37.667 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:37.668 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:37.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:37.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:37.671 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:37.672 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:42.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:42.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:42.674 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:42.723 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:42.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:42.782 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:42.783 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:42.784 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:42.785 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:42.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:42.785 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:42.786 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:47.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:47.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:47.787 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:47.838 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:47.838 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:47.910 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:47.910 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:47.910 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:47.910 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:47.910 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:47.910 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:47.910 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:47.911 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:47.912 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:47.913 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:47.914 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:47.915 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:47.916 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:52.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:52.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:52.918 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:53.037 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:53.037 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:53.037 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:53.038 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:53.039 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:53.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:53.041 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:53.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:53.042 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:53.042 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:53.042 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:53.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:53.042 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:53.043 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:37:58.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:58.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:37:58.044 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:37:58.094 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:37:58.094 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:37:58.151 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:37:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:37:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:37:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:37:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:37:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:37:58.152 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:37:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:37:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:58.153 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:37:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:37:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:37:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:58.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:37:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:37:58.155 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:37:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:58.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:37:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:37:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:58.157 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:37:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:37:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:37:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:37:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:37:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:37:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:37:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:37:58.159 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:37:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:37:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:37:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:37:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:37:58.160 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:37:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:37:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:37:58.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:37:58.161 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:37:58.163 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:03.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:03.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:03.164 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:03.209 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:03.209 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:03.294 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:03.294 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:03.295 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:03.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:03.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:03.298 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:03.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:03.299 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:03.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:08.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:08.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:08.301 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:08.353 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:08.354 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:08.426 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:08.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:08.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:08.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:08.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:08.431 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:08.432 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:13.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:13.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:13.433 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:13.484 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:13.485 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:13.548 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:13.549 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:13.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:13.552 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:13.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:13.553 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:13.553 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:13.553 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:13.553 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:13.553 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:13.554 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:18.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:18.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:18.555 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:18.605 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:18.606 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:18.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:18.689 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:18.690 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:18.691 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:18.692 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:18.693 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:23.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:23.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:23.694 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:23.746 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:23.747 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:23.821 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:23.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:23.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:23.824 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:23.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:23.826 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:23.826 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:23.826 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:23.830 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:28.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:28.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:28.831 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:28.948 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:28.949 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:28.950 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:28.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:28.952 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:28.953 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:33.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:33.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:33.955 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:34.000 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:34.001 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:34.072 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:34.078 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:34.079 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:39.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:39.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:39.080 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:39.189 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:39.189 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:39.190 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:39.191 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:39.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:39.193 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:39.194 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:39.194 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:39.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:44.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:44.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:44.196 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:44.244 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:44.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:44.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:44.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:44.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:44.322 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:44.323 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:49.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:49.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:49.324 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:49.381 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:49.381 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:49.456 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:49.456 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:49.456 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:49.456 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:49.457 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:49.458 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:49.459 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:49.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:49.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:49.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:49.459 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:49.459 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:49.460 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:49.461 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:49.462 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:54.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:54.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:54.464 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:54.514 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:54.580 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:54.580 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:54.581 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:54.582 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:54.583 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:54.584 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:54.585 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:54.586 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:54.586 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:54.587 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:38:59.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:59.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:38:59.588 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:38:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:38:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:59.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:59.740 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:38:59.741 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:59.742 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:38:59.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:38:59.743 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:38:59.743 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:38:59.743 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:38:59.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:38:59.743 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:38:59.744 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:04.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:04.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:04.745 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:04.794 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:04.858 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:04.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:04.860 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:04.861 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279642.023939_142.1", 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:34:02.023939Z", 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:04.862 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:04.863 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:04.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:04.863 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:04.864 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:09.865 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... FAIL 2024-08-21T22:39:09.866 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-17 2024-08-21T22:39:09.887 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-18 2024-08-21T22:39:09.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:09.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:09.961 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:10.005 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:10.006 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:39:10.076 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:39:10.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:10.076 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:39:10.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:39:09.956204Z" 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:10.077 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:39:10.078 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:39:10.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:10.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:10.079 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:10.124 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:10.124 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:10.175 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.176 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.178 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:10.179 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.877930_148.1", 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.877930Z", 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:10.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:10.181 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:10.181 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:10.181 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:10.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:10.181 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:10.182 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724279949.956204_163.1 target=1_1724279949.877930_148.1 2024-08-21T22:39:15.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:15.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:15.183 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:15.233 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:15.298 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:15.298 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:15.299 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:15.300 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:15.301 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:15.302 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:15.303 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:15.305 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:20.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:20.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:20.306 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:20.356 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:20.356 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:20.423 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:20.423 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:20.423 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:20.423 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:20.424 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:20.425 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:20.428 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:20.429 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:25.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:25.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:25.430 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:25.475 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:25.475 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:25.534 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:25.540 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:25.541 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:30.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:30.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:30.542 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:30.651 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:30.651 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:30.652 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:30.653 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:30.654 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:30.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:30.656 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:30.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:30.656 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:30.656 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:30.656 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:30.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:30.656 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:30.657 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:35.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:35.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:35.658 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:35.702 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:35.703 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:35.782 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:35.788 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:40.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:40.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:40.790 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:40.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:40.916 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:40.917 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:40.918 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:45.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:45.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:45.920 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:45.973 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:45.973 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:46.030 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:46.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:46.032 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:46.033 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:46.034 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:46.035 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:46.036 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:51.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:51.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:51.038 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:51.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:51.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:51.138 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:51.139 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:51.140 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:51.141 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:39:56.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:56.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:39:56.142 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:39:56.193 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:39:56.193 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:39:56.313 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:39:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:39:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:39:56.313 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:39:56.314 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:39:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:56.317 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:39:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:39:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:39:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:39:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:39:56.318 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:39:56.318 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:39:56.319 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:01.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:01.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:01.320 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:01.372 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:01.372 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:01.444 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:01.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:01.446 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:01.447 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:01.448 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:01.449 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:01.451 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:06.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:06.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:06.452 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:06.571 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:06.572 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:06.573 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:06.576 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:06.577 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:11.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:11.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:11.578 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:11.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:11.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:11.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:11.709 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:11.710 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:11.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:11.711 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:11.712 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:16.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:16.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:16.713 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:16.763 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:16.764 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:16.820 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:16.821 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:16.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:16.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:16.825 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:16.826 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:21.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:21.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:21.828 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:21.891 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:21.892 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:22.003 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:22.004 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:22.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:22.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:22.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:22.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:22.009 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:22.009 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:22.009 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:22.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:22.009 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:22.010 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:27.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:27.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:27.011 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:27.062 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:27.139 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:27.143 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:27.143 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:27.143 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:27.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:27.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:27.143 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:27.144 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:27.145 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:27.146 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:27.146 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:27.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:27.146 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:27.147 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:32.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:32.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:32.148 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:32.273 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:32.273 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:32.274 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:32.275 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:32.276 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:32.278 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:32.279 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:37.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:37.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:37.281 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:37.332 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:37.333 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:37.396 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:37.396 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:37.396 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:37.397 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:37.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:37.400 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:37.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:37.401 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:37.401 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:37.401 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:37.401 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:37.401 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:37.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:42.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:42.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:42.403 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:42.454 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:42.529 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:42.529 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:42.529 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:42.529 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:42.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:42.530 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:42.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:42.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:42.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:42.534 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:42.535 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:47.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:47.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:47.536 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:47.587 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:47.648 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:47.648 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:47.648 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:47.649 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:47.650 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:47.651 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:47.653 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:47.654 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:52.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:52.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:52.656 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:52.774 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:52.775 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:52.776 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:52.777 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:52.778 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:52.779 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:52.780 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:40:57.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:57.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:40:57.782 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:40:57.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:40:57.827 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:40:57.900 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:40:57.900 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:40:57.900 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:40:57.900 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:57.901 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:57.902 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:57.903 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:40:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:40:57.905 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:40:57.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:02.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:02.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:02.908 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:02.960 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:03.088 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:03.088 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:03.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:03.090 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:03.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:03.092 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:03.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:03.093 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:03.094 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:08.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:08.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:08.096 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:08.142 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:08.142 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:08.215 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:08.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:08.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:08.218 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:08.219 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:08.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:08.221 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:08.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:08.221 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:08.222 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:13.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:13.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:13.223 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:13.373 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:13.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:13.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:13.377 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:13.378 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:13.378 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:13.378 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:13.378 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:13.379 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:18.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:18.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:18.380 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:18.504 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:18.504 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:18.504 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:18.505 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:18.506 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:18.506 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:18.506 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:18.508 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:18.509 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:18.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:18.510 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:18.510 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:18.510 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:18.510 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:18.510 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:18.511 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:23.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:23.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:23.512 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:23.564 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:23.564 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:23.630 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:23.630 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:23.631 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:23.632 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:23.633 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:23.634 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:23.635 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:23.636 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:28.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:28.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:28.638 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:28.688 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:28.689 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:28.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:28.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:28.758 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:28.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:28.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:28.761 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:28.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:28.761 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:28.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:28.761 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:28.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:28.761 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:28.762 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:33.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:33.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:33.763 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:33.814 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:33.815 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:33.892 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:33.893 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:33.894 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:33.895 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:33.896 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:33.897 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:33.898 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:38.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:38.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:38.900 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:39.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:39.014 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:39.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:39.016 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:39.017 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:39.018 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:39.018 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:39.018 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:39.018 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:39.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:44.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:44.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:44.020 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:44.071 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:44.071 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:44.145 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:44.146 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:44.147 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:44.148 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:44.149 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:44.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:44.151 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:44.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:44.151 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:44.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:44.151 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:44.152 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:49.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:49.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:49.153 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:49.263 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:49.263 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:49.264 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:49.265 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:49.266 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:49.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:49.268 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:49.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:49.270 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:49.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:49.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:49.273 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:49.274 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:54.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:54.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:54.275 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:54.407 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:54.407 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:54.408 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:54.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:54.410 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:54.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:54.412 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:54.413 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:41:59.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:59.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:41:59.415 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:41:59.465 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:41:59.466 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:41:59.528 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:41:59.528 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:59.529 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:59.530 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:41:59.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:41:59.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:41:59.533 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:41:59.534 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:04.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:04.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:04.536 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:04.666 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:04.667 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:04.668 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:04.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:04.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:04.671 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:04.672 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:09.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:09.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:09.674 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:09.805 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:09.806 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:09.806 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:09.806 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:09.806 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:09.806 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:09.806 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:09.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:09.807 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:09.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:09.807 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:09.807 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:09.807 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:09.808 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:09.808 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:09.808 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:09.808 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:09.808 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:09.808 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:09.809 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:09.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:09.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:09.809 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:09.809 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:09.809 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:09.810 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:09.810 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:09.810 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:09.810 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:09.810 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:09.811 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:09.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:09.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:09.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:09.811 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:09.811 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:09.812 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:09.812 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:09.812 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:09.812 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:09.812 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:09.813 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:09.813 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:09.813 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:09.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:09.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:09.813 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:09.814 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:09.814 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:09.814 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:09.814 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:09.814 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:09.815 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:09.815 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:09.815 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:09.815 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:09.815 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:09.815 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:09.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:09.816 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:09.816 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:09.816 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:09.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:09.816 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:09.818 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:14.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:14.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:14.820 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:14.923 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:14.923 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:14.923 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:14.923 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:14.923 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:14.924 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:14.925 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:14.926 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:14.927 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:14.928 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:14.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:19.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:19.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:19.931 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:20.051 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:20.053 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:20.054 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:20.055 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:20.056 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:20.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:20.057 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:20.057 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:20.057 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:20.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:20.057 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:20.058 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:25.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:25.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:25.060 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:25.110 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:25.110 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:25.175 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:25.175 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:25.176 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:25.176 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:25.176 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:25.176 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:25.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:25.177 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:25.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:25.177 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:25.177 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:25.177 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:25.178 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:25.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:25.178 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:25.178 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:25.178 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:25.178 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:25.179 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:25.180 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:25.180 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:25.180 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:25.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:25.180 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:25.181 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:25.181 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:25.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:25.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:25.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:25.182 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:25.182 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:25.182 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:25.182 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:25.182 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:25.183 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:25.183 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:25.183 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:25.183 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:25.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:25.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:25.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:25.184 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:25.184 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:25.184 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:25.184 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:25.185 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:25.185 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:25.185 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:25.185 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:25.185 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:25.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:25.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:25.186 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:25.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:25.186 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:25.187 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:25.187 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:25.187 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:25.187 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:25.189 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:30.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:30.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:30.191 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:30.317 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:30.318 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:30.320 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:30.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:30.322 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:30.323 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:35.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:35.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:35.325 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:35.453 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:35.458 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:35.459 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:40.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:40.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:40.460 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:40.511 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:40.511 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:40.573 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:40.573 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:40.573 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:40.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:40.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:40.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:40.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:40.579 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:40.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:45.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:45.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:45.581 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:45.624 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:45.625 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:45.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:45.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:45.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:45.706 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:45.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:45.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:45.709 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:45.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:45.709 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:45.709 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:45.709 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:45.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:45.709 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:45.710 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:50.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:50.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:50.712 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:50.762 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:50.763 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:50.845 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:50.845 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:50.845 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:50.845 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:50.846 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:50.847 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:50.848 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:50.849 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:50.850 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:50.851 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:42:55.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:55.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:42:55.853 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:42:55.904 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:42:55.904 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:55.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:55.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:55.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:42:55.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:42:55.974 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:42:55.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:42:55.975 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:42:55.976 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:00.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:00.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:00.978 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:01.029 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:01.029 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:01.109 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:01.110 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:01.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:01.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:01.113 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:01.114 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:01.115 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:06.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:06.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:06.117 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:06.168 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:06.245 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:06.246 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:06.247 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:06.248 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:06.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:06.250 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:06.251 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:11.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:11.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:11.252 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:11.358 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:11.359 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:11.360 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:11.361 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:11.362 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:11.363 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:11.364 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:11.364 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:11.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:11.364 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:11.365 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:16.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:16.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:16.366 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:16.415 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:16.415 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:16.499 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:16.499 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:16.499 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:16.500 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:16.503 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:16.504 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:16.504 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:16.504 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:16.504 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:16.504 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:16.504 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:16.504 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:16.505 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:16.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:16.507 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:16.507 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:16.507 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:16.507 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:16.507 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:16.509 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:21.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:21.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:21.510 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:21.623 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:21.623 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:21.623 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:21.623 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:21.624 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:21.625 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:21.626 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:21.627 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:21.628 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:21.629 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:26.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:26.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:26.631 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:26.682 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:26.744 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:26.745 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:31.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:31.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:31.746 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:31.796 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:31.797 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:31.866 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:31.866 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:31.867 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:31.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:31.869 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:31.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:31.871 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:31.872 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:36.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:36.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:36.874 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:36.918 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:37.001 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:37.006 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:37.007 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:42.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:42.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:42.009 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:42.059 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:42.118 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:42.118 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:42.118 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:42.118 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:42.119 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:42.120 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:42.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:42.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:42.123 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:42.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:42.123 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:42.123 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:42.123 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:42.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:42.123 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:42.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:47.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:47.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:47.125 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:47.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:47.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:47.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:47.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:47.242 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:47.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:47.243 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:47.243 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:47.243 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:47.243 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:47.243 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:47.244 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:52.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:52.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:52.246 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:52.296 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:52.296 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:52.364 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:52.365 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:52.366 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:52.367 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:52.368 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:52.369 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:43:57.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:57.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:43:57.371 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:43:57.422 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:43:57.422 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:43:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:57.478 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:43:57.479 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:43:57.480 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:43:57.481 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:43:57.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:43:57.482 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:43:57.483 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:02.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:02.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:02.484 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:02.535 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:02.604 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:02.605 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:07.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:07.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:07.607 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:07.651 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:07.652 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:07.714 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:07.715 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:07.716 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:07.717 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:07.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:07.719 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:07.720 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:12.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:12.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:12.722 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:12.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:12.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:12.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:12.872 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724279949.956204_163.1", 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:39:09.956204Z", 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:12.873 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:12.874 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:12.874 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:12.874 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:12.875 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:17.876 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... FAIL 2024-08-21T22:44:17.877 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-19 2024-08-21T22:44:17.905 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-20 2024-08-21T22:44:17.942 INFO:rgw_multi.tests:create bucket zone=b1 name=qlihov-21 2024-08-21T22:44:18.056 INFO:rgw_multi.tests:create bucket zone=b2 name=qlihov-22 2024-08-21T22:44:18.127 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2024-08-21T22:44:18.128 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-23 2024-08-21T22:44:18.154 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-24 2024-08-21T22:44:18.185 INFO:rgw_multi.tests:create bucket zone=b1 name=qlihov-25 2024-08-21T22:44:18.249 INFO:rgw_multi.tests:create bucket zone=b2 name=qlihov-26 2024-08-21T22:44:18.487 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2024-08-21T22:44:18.487 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-27 2024-08-21T22:44:18.507 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-28 2024-08-21T22:44:18.534 INFO:rgw_multi.tests:create bucket zone=b1 name=qlihov-29 2024-08-21T22:44:18.589 INFO:rgw_multi.tests:create bucket zone=b2 name=qlihov-30 2024-08-21T22:44:21.748 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2024-08-21T22:44:21.748 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-31 2024-08-21T22:44:21.774 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-32 2024-08-21T22:44:21.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:21.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:21.828 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:21.884 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:21.884 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:21.953 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:44:21.968 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:44:21.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:21.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:21.970 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:21.970 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:44:21.794805Z" 2024-08-21T22:44:21.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:21.970 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:44:21.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:44:21.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:21.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:21.971 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:22.023 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:22.023 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:22.119 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:22.119 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:22.119 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:22.119 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:22.119 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:22.119 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:22.119 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:22.120 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:22.121 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:22.122 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:22.123 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:22.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:22.125 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:22.125 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:22.125 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:22.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:22.125 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:22.126 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:27.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:27.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:27.128 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:27.250 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:27.250 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:27.250 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:27.250 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:27.250 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:27.250 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:27.251 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:27.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:27.251 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:27.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:27.251 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:27.251 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:27.251 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:27.252 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:27.253 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:27.254 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:27.255 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:27.256 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:32.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:32.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:32.258 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:32.379 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:32.379 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:32.379 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:32.380 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:32.381 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:32.382 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:32.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:32.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:32.385 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:32.386 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:37.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:37.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:37.387 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:37.435 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:37.491 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:37.491 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:37.492 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:37.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:37.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:37.495 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:37.496 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:37.497 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:42.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:42.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:42.499 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:42.550 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:42.626 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:42.626 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:42.627 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:42.628 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:42.629 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:42.630 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:42.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:42.631 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:42.631 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:42.631 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:42.631 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:42.631 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:42.632 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:47.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:47.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:47.633 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:47.686 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:47.758 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:47.758 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:47.758 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:47.758 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:47.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:47.760 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:47.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:47.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:47.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:47.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:47.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:47.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:47.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:47.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:47.764 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:47.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:47.765 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:47.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:52.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:52.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:52.767 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:52.820 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:52.820 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:52.885 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:52.885 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:52.885 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:52.886 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:52.887 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:52.888 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:52.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:52.890 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:52.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:52.891 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:52.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:52.891 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:52.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:52.891 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:52.892 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:44:57.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:57.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:44:57.894 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:44:57.946 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:44:57.946 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:44:58.025 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:58.026 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:58.027 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:58.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:44:58.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:44:58.030 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:44:58.031 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:03.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:03.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:03.032 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:03.138 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:03.138 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:03.139 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:03.140 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:03.141 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:03.142 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:03.143 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:03.144 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:08.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:08.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:08.146 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:08.197 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:08.197 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:08.275 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:08.275 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:08.276 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:08.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:08.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:08.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:08.280 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:08.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:08.280 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:08.280 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:08.280 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:08.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:08.280 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:08.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:13.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:13.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:13.282 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:13.333 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:13.334 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:13.409 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:13.409 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:13.409 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:13.409 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:13.409 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:13.409 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:13.410 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:13.411 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:13.412 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:13.413 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:13.414 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:13.415 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:13.415 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:13.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:13.415 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:13.416 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:18.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:18.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:18.417 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:18.469 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:18.469 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:18.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:18.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:18.529 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:18.530 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:18.531 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:18.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:18.532 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:18.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:18.532 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:18.533 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:23.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:23.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:23.534 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:23.660 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:23.660 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:23.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:23.660 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:23.660 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:23.660 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:23.661 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:23.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:23.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:23.664 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:23.665 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:23.666 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:28.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:28.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:28.668 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:28.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:28.719 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:28.777 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:28.777 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:28.777 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:28.778 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:28.779 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:28.780 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:28.781 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:28.782 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:28.783 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:33.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:33.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:33.785 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:33.915 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:33.915 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:33.915 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:33.915 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:33.915 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:33.915 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:33.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:33.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:33.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:33.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:33.920 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:33.920 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:33.920 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:33.920 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:33.920 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:33.921 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:38.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:38.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:38.923 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:38.973 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:38.974 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:39.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:39.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:39.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:39.050 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:39.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:39.052 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:39.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:39.053 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:39.054 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:44.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:44.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:44.056 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:44.165 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:44.165 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:44.166 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:44.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:44.168 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:44.169 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:44.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:44.170 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:44.171 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:49.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:49.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:49.172 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:49.282 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:49.283 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:49.285 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:49.286 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:49.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:49.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:49.289 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:49.290 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:49.291 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:49.293 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:49.293 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:49.294 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:54.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:54.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:54.296 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:54.347 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:54.347 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:54.427 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:54.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:54.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:54.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:54.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:54.432 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:54.433 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:45:59.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:59.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:45:59.435 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:45:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:45:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:59.557 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:59.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:59.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:45:59.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:45:59.561 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:45:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:45:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:45:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:45:59.562 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:45:59.563 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:04.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:04.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:04.564 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:04.614 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:04.615 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:04.675 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:04.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:04.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:04.678 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:04.679 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:04.680 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:09.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:09.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:09.682 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:09.781 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:09.782 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:09.783 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:09.784 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:09.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:09.786 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:09.787 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:14.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:14.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:14.788 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:14.839 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:14.839 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:14.913 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:14.914 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:14.915 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:14.916 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:14.917 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:14.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:14.919 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:14.919 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:19.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:19.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:19.921 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:19.972 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:19.972 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:20.047 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:20.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:20.049 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:20.050 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:20.051 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:20.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:20.052 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:20.053 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:25.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:25.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:25.055 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:25.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:25.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:25.166 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:25.166 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:25.166 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:25.166 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:25.167 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:25.168 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:25.169 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:25.170 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:25.171 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:25.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:25.172 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:25.172 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:25.172 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:25.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:25.172 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:25.173 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:30.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:30.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:30.174 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:30.226 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:30.226 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:30.307 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:30.308 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:30.309 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:30.310 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:30.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:30.312 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:30.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:35.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:35.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:35.315 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:35.421 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:35.422 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:40.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:40.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:40.424 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:40.469 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:40.527 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:40.528 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:40.529 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:40.530 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:40.531 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:40.532 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:45.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:45.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:45.534 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:45.586 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:45.669 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:45.669 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:45.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:45.671 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:45.672 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:45.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:45.674 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:45.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:50.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:50.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:50.677 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:50.727 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:50.796 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:50.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:50.798 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:50.799 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:50.800 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:50.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:50.802 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:50.802 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:46:55.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:55.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:46:55.804 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:46:55.854 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:46:55.855 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:46:55.920 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:46:55.920 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:46:55.920 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:46:55.922 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:46:55.923 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:46:55.924 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:46:55.925 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:46:55.926 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:46:55.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:46:55.926 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:46:55.927 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:00.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:00.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:00.928 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:01.057 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:01.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:01.059 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:01.060 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:01.061 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:01.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:01.062 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:01.063 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:06.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:06.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:06.064 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:06.115 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:06.185 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:06.186 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:06.187 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:06.188 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:06.190 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:06.191 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:11.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:11.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:11.192 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:11.242 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:11.242 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:11.299 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:11.299 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:11.299 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:11.299 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:11.299 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:11.300 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:11.301 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:11.302 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:11.303 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:11.304 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:11.304 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:11.304 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:11.305 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:16.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:16.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:16.307 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:16.358 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:16.359 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:16.426 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:16.426 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:16.426 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:16.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:16.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:16.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:16.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:16.431 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:16.432 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:21.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:21.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:21.434 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:21.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:21.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:21.569 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:21.570 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:21.571 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:21.572 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:26.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:26.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:26.573 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:26.701 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:26.701 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:26.701 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:26.701 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:26.702 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:26.703 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:26.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:26.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:26.706 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:26.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:31.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:31.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:31.708 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:31.755 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:31.756 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:31.818 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:31.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:31.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:31.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:31.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:31.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:31.824 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:31.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:36.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:36.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:36.827 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:36.890 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:36.890 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:37.001 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:37.001 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:37.002 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:37.003 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:37.004 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:37.005 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:37.006 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:37.007 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:37.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:37.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:37.007 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:37.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:37.007 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:37.008 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:37.008 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:37.008 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:37.008 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:37.010 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:42.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:42.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:42.012 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:42.057 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:42.137 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:42.138 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:42.139 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:42.140 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:42.141 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:42.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:42.143 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:42.144 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:47.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:47.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:47.145 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:47.284 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:47.285 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:47.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:47.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:47.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:47.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:47.290 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:47.290 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:47.290 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:47.291 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:52.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:52.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:52.292 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:52.402 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:52.402 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:52.402 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:52.402 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:52.402 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:52.402 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:52.402 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:52.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:52.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:52.405 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:52.406 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:52.407 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:52.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:52.407 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:52.408 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:47:57.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:57.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:47:57.409 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:47:57.459 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:47:57.459 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:47:57.538 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:47:57.538 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:47:57.539 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:57.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:57.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:47:57.542 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:47:57.543 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:47:57.544 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:02.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:02.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:02.546 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:02.597 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:02.598 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:02.671 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:02.677 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:02.678 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:07.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:07.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:07.679 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:07.728 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:07.728 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:07.784 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:07.785 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:07.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:07.787 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:07.788 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:07.789 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:07.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:12.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:12.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:12.792 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:12.856 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:12.856 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:12.949 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:12.950 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:12.951 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:12.952 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:12.953 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:12.954 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:12.954 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:12.955 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:17.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:17.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:17.956 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:18.084 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:18.085 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:18.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:18.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:18.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:18.089 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:18.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:18.090 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:18.091 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:23.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:23.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:23.093 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:23.214 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:23.214 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:23.214 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:23.214 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:23.214 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:23.215 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:23.216 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:23.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:23.218 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:23.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:23.219 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:23.219 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:23.219 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:23.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:23.219 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:23.220 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:28.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:28.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:28.221 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:28.272 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:28.352 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:28.352 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:28.352 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:28.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:28.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:28.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:28.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:28.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:28.357 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:28.358 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:33.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:33.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:33.360 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:33.409 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:33.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:33.479 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:33.480 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:33.481 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:33.482 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:33.483 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:33.483 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:33.484 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:38.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:38.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:38.485 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:38.534 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:38.591 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:38.592 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:38.593 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:38.594 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:38.595 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:38.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:38.596 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:38.597 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:43.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:43.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:43.598 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:43.651 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:43.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:43.723 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:43.724 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:43.725 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:43.726 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:43.727 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:48.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:48.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:48.729 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:48.781 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:48.862 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:48.862 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:48.862 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:48.862 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:48.863 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:48.864 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:48.865 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:48.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:48.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:48.868 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:48.868 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:48.868 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:48.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:48.868 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:48.869 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:53.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:53.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:53.870 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:53.980 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:53.980 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:53.981 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:53.982 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:53.983 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:53.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:53.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:53.985 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:53.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:48:58.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:58.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:48:58.987 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:48:59.033 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:48:59.033 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:48:59.108 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:48:59.108 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:48:59.108 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:48:59.108 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:48:59.108 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:48:59.108 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:48:59.109 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:59.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:59.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:48:59.112 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:48:59.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:48:59.113 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:48:59.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:48:59.113 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:48:59.114 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:04.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:04.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:04.115 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:04.184 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:04.184 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:04.292 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:04.292 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:04.292 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:04.292 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:04.293 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:04.294 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:04.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:04.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:04.297 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:04.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:04.297 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:04.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:04.297 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:04.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:04.297 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:04.298 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:09.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:09.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:09.299 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:09.351 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:09.351 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:09.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:09.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:09.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:09.446 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:09.447 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:09.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:09.448 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:09.449 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:14.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:14.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:14.450 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:14.502 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:14.571 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:14.572 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:14.573 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:14.574 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:14.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:14.576 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:14.577 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:19.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:19.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:19.579 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:19.706 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:19.707 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:19.708 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:19.709 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:19.710 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:19.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:19.712 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:19.712 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:19.712 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:19.712 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:19.712 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:19.713 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:24.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:24.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:24.715 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:24.846 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:24.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:24.848 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:24.849 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280261.794805_254.1", 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:44:21.794805Z", 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:24.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:24.851 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:24.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:24.851 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:24.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:24.851 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:24.852 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:29.854 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2024-08-21T22:49:29.855 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-33 2024-08-21T22:49:29.883 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-34 2024-08-21T22:49:29.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:29.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:29.957 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:30.007 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:49:30.057 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:49:30.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:49:29.906815Z" 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:30.059 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:49:30.060 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:49:30.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:30.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:30.061 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:30.104 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:30.104 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:30.177 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.869622_260.1", 2024-08-21T22:49:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.869622Z", 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:30.182 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:30.183 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280569.906815_267.1 target=1_1724280569.869622_260.1 2024-08-21T22:49:35.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:35.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:35.185 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:35.236 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:35.236 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:35.321 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:49:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:35.326 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:35.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:40.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:40.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:40.328 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:40.379 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:40.380 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:40.446 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:40.446 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:40.446 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:40.446 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:40.447 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:40.447 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:40.447 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:40.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:40.447 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:40.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:40.448 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:40.448 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:40.448 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:40.448 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:40.448 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:40.448 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:40.449 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:40.449 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:40.449 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:40.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:40.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:40.449 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:40.449 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:49:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:40.455 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:40.456 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:40.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:45.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:45.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:45.458 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:45.509 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:45.510 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:45.573 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:45.573 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:45.573 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:45.574 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:45.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:45.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:45.578 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:45.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:45.578 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:45.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:45.578 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:45.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:45.578 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:45.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:50.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:50.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:50.581 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:50.697 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:50.697 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:50.698 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:50.699 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:50.700 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:49:50.701 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:50.702 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:50.703 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:49:55.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:55.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:49:55.705 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:49:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:49:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:49:55.820 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:49:55.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:55.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:55.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:49:55.824 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:49:55.825 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:49:55.826 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:49:55.826 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:49:55.826 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:49:55.826 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:49:55.827 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:00.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:00.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:00.828 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:00.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:00.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:00.943 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:00.947 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:00.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:00.948 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:00.949 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:05.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:05.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:05.950 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:06.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:06.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:06.080 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:06.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:06.082 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:06.083 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:11.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:11.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:11.084 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:11.136 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:11.136 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:11.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:11.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:11.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:11.197 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:11.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:11.199 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:11.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:11.199 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:11.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:11.199 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:11.200 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:16.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:16.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:16.202 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:16.252 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:16.252 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:16.339 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:16.339 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:16.339 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:16.339 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:16.339 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:16.339 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:16.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:16.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:16.344 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:16.345 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:21.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:21.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:21.346 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:21.391 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:21.391 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:21.448 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:21.450 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:21.451 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:21.452 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:21.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:21.453 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:21.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:21.453 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:21.453 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:21.453 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:21.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:21.454 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:21.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:26.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:26.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:26.456 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:26.570 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:26.571 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:26.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:26.573 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:26.574 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:31.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:31.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:31.576 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:31.627 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:31.628 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:31.703 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:31.704 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:31.708 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:31.709 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:36.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:36.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:36.710 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:36.761 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:36.761 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:36.824 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:36.824 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:36.824 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:36.824 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:36.824 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:36.824 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:36.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:36.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:36.826 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:36.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:36.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:36.829 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:36.829 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:36.829 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:36.829 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:36.829 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:36.830 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:41.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:41.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:41.832 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:41.882 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:41.953 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:41.954 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:41.955 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:41.956 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:41.957 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:41.958 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:41.959 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:46.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:46.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:46.961 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:47.072 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:47.072 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:47.072 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:47.072 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:47.073 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:47.074 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:47.075 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:47.076 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:47.077 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:47.078 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:52.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:52.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:52.080 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:52.131 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:52.132 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:52.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:52.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:52.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:52.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:52.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:52.224 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:52.225 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:50:57.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:57.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:50:57.227 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:50:57.279 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:50:57.279 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:50:57.408 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:50:57.408 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:50:57.408 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:50:57.408 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:50:57.409 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:57.410 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:50:57.412 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:50:57.413 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:50:57.415 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:02.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:02.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:02.416 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:02.466 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:02.467 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:02.545 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:02.551 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:02.552 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:07.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:07.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:07.553 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:07.605 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:07.606 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:07.665 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:07.665 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:07.665 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:07.665 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:07.665 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:07.666 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:07.667 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:07.668 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:07.669 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:07.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:07.671 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:07.672 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:12.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:12.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:12.674 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:12.726 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:12.726 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:12.809 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:12.809 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:12.811 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:12.812 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:12.813 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:12.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:12.814 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:12.814 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:12.814 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:12.814 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:12.814 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:12.815 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:17.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:17.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:17.817 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:17.869 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:17.928 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:17.929 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:17.931 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:17.932 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:17.933 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:17.934 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:22.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:22.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:22.936 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:22.986 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:23.058 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:23.058 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:23.058 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:23.058 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:23.058 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:23.058 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:23.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:23.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:23.061 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:23.062 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:23.063 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:23.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:23.063 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:23.064 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:28.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:28.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:28.065 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:28.197 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:28.202 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:28.202 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:28.202 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:28.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:28.202 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:28.203 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:33.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:33.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:33.204 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:33.318 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:33.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:33.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:33.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:33.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:33.321 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:33.322 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:33.322 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:33.322 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:33.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:33.322 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:33.323 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:33.323 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:33.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:33.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:33.323 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:33.323 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:33.324 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:33.324 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:33.324 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:33.324 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:33.324 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:33.324 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:33.325 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:33.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:33.326 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:33.327 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:38.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:38.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:38.328 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:38.376 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:38.376 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:38.456 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:38.457 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:38.458 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:38.459 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:38.460 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:38.461 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:38.462 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:43.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:43.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:43.464 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:43.572 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:43.572 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:43.572 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:43.572 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:43.573 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:43.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:43.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:43.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:43.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:43.578 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:43.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:48.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:48.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:48.580 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:48.631 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:48.718 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:48.718 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:48.718 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:48.718 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:48.718 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:48.718 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:48.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:48.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:48.721 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:48.722 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:48.723 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:48.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:48.723 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:48.724 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:53.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:53.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:53.725 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:53.775 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:53.775 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:53.851 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:53.851 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:53.851 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:53.851 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:53.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:53.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:53.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:53.855 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:53.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:53.856 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:53.856 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:53.856 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:53.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:53.856 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:53.857 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:51:58.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:58.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:51:58.858 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:51:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:51:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:58.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:58.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:58.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:51:58.975 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:51:58.976 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:51:58.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:51:58.977 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:51:58.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:03.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:03.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:03.979 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:04.031 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:04.031 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:04.105 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:04.106 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:04.107 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:04.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:04.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:04.110 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:04.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:04.111 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:04.112 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:09.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:09.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:09.113 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:09.164 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:09.165 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:09.246 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:09.246 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:09.246 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:09.246 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:09.246 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:09.247 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:09.248 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:09.249 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:09.250 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:09.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:09.252 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:09.252 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:09.252 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:09.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:09.252 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:09.253 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:14.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:14.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:14.254 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:14.359 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:14.359 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:14.359 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:14.359 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:14.360 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:14.361 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:14.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:14.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:14.364 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:14.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:14.364 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:14.364 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:14.364 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:14.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:14.364 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:14.365 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:19.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:19.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:19.366 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:19.418 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:19.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:19.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:19.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:19.490 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:19.491 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:19.492 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:24.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:24.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:24.493 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:24.545 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:24.545 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:24.605 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:24.606 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:24.607 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:24.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:24.609 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:24.610 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:24.611 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:24.611 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:24.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:24.611 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:24.612 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:29.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:29.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:29.614 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:29.664 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:29.664 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:29.736 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:29.737 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:29.738 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:29.739 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:29.740 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:29.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:34.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:34.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:34.742 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:34.793 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:34.793 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:34.856 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:34.862 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:34.864 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:39.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:39.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:39.865 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:39.916 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:39.916 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:39.975 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:39.975 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:39.975 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:39.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:39.977 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:39.978 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:39.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:39.980 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:39.981 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:44.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:44.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:44.983 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:45.101 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:45.101 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:45.101 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:45.101 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:45.101 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:45.101 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:45.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:45.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:45.104 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:45.105 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:45.106 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:45.106 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:45.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:45.106 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:45.108 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:50.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:50.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:50.109 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:50.156 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:50.156 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:50.215 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:50.215 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:50.216 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:50.217 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:50.218 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:50.219 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:50.220 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:50.221 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:52:55.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:55.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:52:55.223 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:52:55.268 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:52:55.268 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:52:55.344 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:52:55.344 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:55.345 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:55.346 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:55.347 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:52:55.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:55.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:55.350 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:52:55.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:52:55.350 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:52:55.350 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:52:55.350 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:52:55.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:52:55.350 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:52:55.351 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:00.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:00.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:00.352 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:00.403 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:00.404 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:00.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:00.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:00.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:00.489 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:00.490 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:00.491 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:00.492 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:05.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:05.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:05.495 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:05.547 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:05.635 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:05.635 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:05.635 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:05.635 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:05.635 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:05.636 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:05.637 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:05.638 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:05.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:05.640 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:05.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:05.640 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:05.640 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:05.640 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:05.640 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:05.640 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:05.641 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:10.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:10.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:10.642 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:10.693 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:10.693 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:10.764 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:10.764 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:10.764 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:10.764 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:10.764 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:10.764 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:10.765 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:10.766 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:10.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:10.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:10.769 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:10.769 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:10.769 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:10.769 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:10.769 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:10.770 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:15.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:15.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:15.772 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:15.822 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:15.822 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:15.888 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:15.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:15.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:15.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:15.892 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:15.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:15.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:15.894 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:15.896 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:20.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:20.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:20.898 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:20.948 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:20.948 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:21.018 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:21.018 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:21.018 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:21.019 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:21.020 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:21.021 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:21.022 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:21.023 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:21.024 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:26.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:26.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:26.025 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:26.076 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:26.077 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:26.142 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:26.143 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:26.144 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:26.145 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:26.147 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:26.149 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:31.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:31.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:31.150 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:31.203 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:31.203 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:31.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:31.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:31.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:31.285 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:31.285 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:31.285 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:31.285 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:31.285 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:31.286 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:36.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:36.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:36.287 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:36.338 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:36.429 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:36.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:36.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:36.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:36.433 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:36.434 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:36.435 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:41.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:41.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:41.437 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:41.488 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:41.488 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:41.562 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:41.562 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:41.563 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:41.564 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:41.565 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:41.566 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:41.567 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:41.569 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:46.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:46.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:46.570 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:46.616 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:46.670 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:46.670 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:46.671 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:46.672 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:46.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:46.674 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:46.675 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:46.677 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:51.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:51.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:51.679 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:51.730 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:51.730 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:51.816 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:51.816 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:51.817 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:51.818 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:51.819 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:51.820 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:51.821 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:51.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:53:56.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:56.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:53:56.824 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:53:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:53:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:53:56.959 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:53:56.959 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:53:56.959 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:53:56.959 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:56.960 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:56.961 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:53:56.962 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:56.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:56.964 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:53:56.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:53:56.964 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:53:56.964 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:53:56.964 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:53:56.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:53:56.964 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:53:56.965 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:01.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:01.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:01.967 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:54:02.019 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:02.019 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:02.087 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:02.088 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:07.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:07.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:07.090 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:54:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:07.289 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:07.289 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:54:07.289 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:54:07.289 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:54:07.289 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:07.290 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:54:07.291 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:54:07.292 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:07.294 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:07.295 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:12.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:12.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:12.296 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:54:12.341 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:12.341 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:12.402 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:12.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:12.404 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:12.405 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:12.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:12.407 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:54:12.407 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:54:12.407 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:54:12.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:12.407 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:12.408 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:17.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:17.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:17.409 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:54:17.472 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:17.472 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:54:17.571 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:17.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:17.573 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:17.574 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:17.575 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:17.576 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:22.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:22.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:22.578 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:54:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:22.696 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:22.696 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:54:22.696 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:54:22.696 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:54:22.696 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:22.697 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:22.698 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:22.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:22.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:54:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:54:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:54:22.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:22.701 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:22.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:27.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:27.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:27.704 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:54:27.754 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:27.755 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:27.831 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:54:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:54:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:54:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:54:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:27.835 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:27.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:27.836 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:54:27.836 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:54:27.836 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:54:27.836 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:27.836 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:27.837 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:32.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:32.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:54:32.839 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:54:32.890 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:32.890 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:32.966 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:54:32.967 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:32.968 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:32.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:54:32.970 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:32.971 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:32.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:32.972 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:54:32.972 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:54:32.972 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:54:32.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:32.972 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:32.973 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:54:37.976 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... rgw_multi.tests.test_datalog_autotrim ... FAIL 2024-08-21T22:54:37.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:54:37.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:54:37.977 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T22:54:38.025 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:38.025 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a2", 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "a2.rgw.meta:root", 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "a2.rgw.control", 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "a2.rgw.log:gc", 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "a2.rgw.log:lc", 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "a2.rgw.log", 2024-08-21T22:54:38.056 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "a2.rgw.log:intent", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "a2.rgw.log:usage", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "a2.rgw.meta:roles", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "a2.rgw.log:reshard", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "a2.rgw.otp", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "SFQCSPIH466OOI28", 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "Y6NKkqiemixittNnTBCkGQzIGFnED8DL" 2024-08-21T22:54:38.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "a2.rgw.buckets.index", 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "a2.rgw.buckets.data", 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "compression_type": "random" 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.058 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "a2.rgw.log:notif" 2024-08-21T22:54:38.059 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:38.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:54:38.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T22:54:38.061 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T22:54:38.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:54:38.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:54:38.153 INFO:teuthology.orchestra.run.smithi116.stderr:Sending period to new master zone 5407971e-5f7f-4ba8-8316-03f497e31665 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 5, 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: "predecessor_uuid": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": [], 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: "period_map": { 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: "zonegroups": [ 2024-08-21T22:54:38.204 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a", 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.205 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a2", 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8001" 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "false", 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a1", 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T22:54:38.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T22:54:38.208 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "b", 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "false", 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:54:38.209 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b2", 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8001" 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T22:54:38.210 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b1", 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T22:54:38.211 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T22:54:38.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: "short_zone_ids": [ 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 1441323143 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 4021227367 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 943360455 2024-08-21T22:54:38.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 3475667924 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zonegroup": "a", 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "period_config": { 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:54:38.228 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "user_ratelimit": { 2024-08-21T22:54:38.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_ratelimit": { 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "anonymous_ratelimit": { 2024-08-21T22:54:38.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_name": "test-realm", 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:54:38.231 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:54:38.233 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2024-08-21T22:55:38.233 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-35 2024-08-21T22:55:38.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:38.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:38.272 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:55:38.320 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:55:38.320 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:55:38.374 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T22:55:38.399 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T22:55:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:55:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T22:55:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:55:38.254894Z" 2024-08-21T22:55:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:38.401 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T22:55:38.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T22:55:38.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:38.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:38.402 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:55:38.446 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:55:38.446 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:55:38.519 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:55:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:55:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:55:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:55:38.519 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:38.520 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:38.521 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:55:38.523 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:38.524 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:55:38.525 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:55:43.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:43.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:43.527 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:55:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:55:43.579 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:55:43.653 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:55:43.653 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:55:43.653 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:55:43.653 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:55:43.654 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:43.655 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:55:43.656 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:43.658 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:55:43.659 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:55:48.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:48.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:48.660 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:55:48.711 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:55:48.711 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:48.793 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:48.794 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:48.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:48.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:48.797 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:55:48.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:48.798 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:55:48.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:55:48.798 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:55:48.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:48.798 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:55:48.799 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:55:53.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:53.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:53.801 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:55:53.851 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:55:53.851 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:55:53.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:53.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:53.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:53.912 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:53.913 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:53.914 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:55:53.916 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:55:58.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:58.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:55:58.917 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:55:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:55:58.967 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:55:59.026 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:55:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:55:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:55:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:55:59.026 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:59.027 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:55:59.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:55:59.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:55:59.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:55:59.031 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:55:59.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:04.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:04.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:04.033 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:04.085 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:04.085 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:04.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:04.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:04.173 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:04.174 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:04.175 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:04.176 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:04.176 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:04.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:04.176 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:04.177 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:09.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:09.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:09.178 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:09.294 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:09.294 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:09.294 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:09.294 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:09.294 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:09.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:09.296 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:09.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:09.298 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:09.299 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:09.300 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:14.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:14.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:14.302 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:14.352 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:14.352 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:14.432 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:14.433 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:14.434 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:14.435 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:14.436 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:14.437 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:14.438 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:19.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:19.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:19.440 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:19.492 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:19.492 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:19.559 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:19.559 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:19.559 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:19.559 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:19.559 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:19.560 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:19.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:19.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:19.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:19.564 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:19.565 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:24.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:24.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:24.567 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:24.682 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:24.682 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:24.682 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:24.683 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:24.684 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:24.685 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:56:24.686 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:24.687 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:24.688 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:29.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:29.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:29.690 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:29.741 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:29.742 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:29.819 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:29.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:29.821 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:29.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:56:29.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:29.824 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:29.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:29.825 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:29.826 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:34.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:34.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:34.827 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:34.878 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:34.939 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280569.906815_267.1", 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:49:29.906815Z", 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:34.944 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:34.945 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280569.906815_267.1 2024-08-21T22:56:39.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:39.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:39.947 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:39.991 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:39.992 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:40.050 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:40.050 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:40.050 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:40.051 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:40.052 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:40.053 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:40.054 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.247773_289.1", 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.247773Z", 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:40.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:40.056 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:40.057 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724280938.254894_297.1 target=1_1724280938.247773_289.1 2024-08-21T22:56:45.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:45.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:45.058 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:45.107 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:45.109 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:45.184 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:45.184 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:45.184 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:45.184 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:45.184 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:45.184 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:45.185 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:45.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:45.187 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:45.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:45.189 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:45.189 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:45.189 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:45.189 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:45.189 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:45.190 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:56:50.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:50.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:50.191 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:50.242 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:50.242 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:50.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:50.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:50.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:50.313 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:50.314 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:50.315 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:50.315 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:50.315 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:50.315 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:50.316 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:56:55.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:55.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:56:55.317 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:56:55.371 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:56:55.372 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:55.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:55.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:56:55.446 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:55.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:56:55.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:56:55.449 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:56:55.450 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:00.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:00.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:00.452 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:00.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:00.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:00.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:00.578 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:00.579 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:00.580 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:05.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:05.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:05.582 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:05.691 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:05.691 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:05.692 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:05.693 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:05.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:05.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:05.696 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:05.697 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:10.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:10.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:10.698 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:10.751 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:10.751 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:10.820 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:10.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:10.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:10.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:10.824 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:10.825 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:10.826 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:10.827 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:10.828 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:15.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:15.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:15.829 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:15.945 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:15.945 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:15.945 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:15.945 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:15.945 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:15.946 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:15.947 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:15.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:15.949 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:15.950 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:15.950 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:15.951 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:20.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:20.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:20.952 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:21.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:21.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:21.084 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:21.084 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:21.084 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:21.084 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:21.084 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:21.085 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:21.085 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:21.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:21.085 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:21.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:21.085 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:21.085 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:21.086 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:21.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:21.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:21.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:21.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:21.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:21.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:21.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:21.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:21.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:21.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:21.091 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:21.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:21.091 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:21.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:21.091 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:21.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:21.091 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:21.092 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:26.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:26.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:26.093 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:26.214 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:26.214 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:26.214 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:26.215 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:26.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:26.217 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:26.218 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:26.219 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:26.221 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:31.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:31.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:31.222 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:31.268 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:31.268 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:31.348 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:31.348 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:31.349 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:31.349 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:31.349 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:31.349 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:31.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:31.350 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:31.351 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:31.352 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:31.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:31.354 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:31.355 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:36.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:36.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:36.357 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:36.464 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:36.464 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:36.464 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:36.464 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:36.464 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:36.464 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:36.464 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:36.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:36.466 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:36.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:36.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:36.469 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:36.469 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:36.469 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:36.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:36.469 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:36.470 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:41.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:41.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:41.471 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:41.588 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:41.588 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:41.589 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:41.590 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:41.592 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:41.593 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:41.593 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:41.593 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:46.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:46.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:46.594 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:46.645 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:46.645 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:46.724 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:46.725 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:46.726 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:46.727 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:46.728 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:46.728 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:46.728 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:46.728 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:46.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:46.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:46.728 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:46.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:46.729 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:46.729 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:46.729 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:46.729 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:46.729 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:46.730 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:51.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:51.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:51.731 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:51.782 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:51.851 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:51.851 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:51.851 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:51.851 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:51.851 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:51.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:51.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:51.854 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:51.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:51.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:51.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:51.856 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:51.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:51.856 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:51.857 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:51.857 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:51.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:51.858 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:51.859 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:57:56.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:56.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:57:56.860 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:57:56.904 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:57:56.904 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:57:56.996 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:57:56.996 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:57:56.996 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:57:56.996 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:57:56.996 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:56.997 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:57:56.998 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:56.999 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:57:57.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:57:57.001 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:57:57.002 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:02.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:02.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:02.004 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:02.048 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:02.049 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:02.115 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:02.115 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:07.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:07.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:07.116 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:07.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:07.167 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:07.240 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:07.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:07.242 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:07.243 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:07.244 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:07.245 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:07.246 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:07.248 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:12.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:12.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:12.249 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:12.358 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:12.358 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:12.358 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:12.359 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:12.360 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:12.361 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:12.362 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:12.363 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:12.364 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:17.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:17.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:17.366 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:17.417 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:17.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:17.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:17.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:17.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:17.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:17.495 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:17.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:17.495 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:17.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:17.495 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:17.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:22.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:22.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:22.498 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:22.547 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:22.547 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:22.608 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:22.608 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:22.608 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:22.608 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:22.608 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:22.608 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:22.608 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:22.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:22.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:22.611 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:22.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:22.613 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:22.613 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:22.613 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:22.613 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:22.613 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:22.614 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:27.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:27.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:27.616 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:27.661 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:27.661 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:27.716 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:27.716 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:27.716 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:27.716 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:27.716 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:27.716 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:27.716 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:27.717 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:27.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:27.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:27.721 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:27.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:27.721 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:27.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:27.721 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:27.722 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:32.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:32.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:32.724 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:32.849 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:32.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:32.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:32.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:32.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:32.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:32.855 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:32.855 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:32.855 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:32.856 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:37.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:37.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:37.857 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:37.966 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:37.967 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:37.968 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:37.968 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:37.968 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:37.968 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:37.968 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:37.968 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:37.969 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:37.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:37.971 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:37.973 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:42.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:42.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:42.975 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:43.026 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:43.097 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:43.097 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:43.097 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:43.097 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:43.097 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:43.097 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:43.098 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:43.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:43.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:43.101 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:43.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:43.102 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:43.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:43.102 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:43.103 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:48.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:48.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:48.104 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:48.235 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:48.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:48.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:48.239 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:48.240 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:48.240 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:48.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:48.240 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:48.241 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:53.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:53.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:53.242 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:53.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:53.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:53.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:53.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:53.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:53.348 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:53.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:53.348 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:53.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:53.348 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:53.349 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:58:58.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:58.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:58:58.351 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:58:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:58:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:58:58.528 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:58.529 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:58.530 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:58:58.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:58:58.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:58:58.533 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:58:58.534 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:03.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:03.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:03.536 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:03.657 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:03.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:03.659 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:03.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:03.662 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:03.663 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:08.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:08.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:08.664 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:08.715 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:08.715 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:08.798 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:08.798 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:08.798 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:08.798 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:08.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:08.800 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:08.801 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:08.802 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:08.802 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:08.802 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:08.802 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:08.802 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:08.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:08.803 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:08.804 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:08.804 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:08.804 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:08.804 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:08.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:08.805 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:08.805 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:08.805 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:08.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:08.805 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:08.806 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:13.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:13.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:13.808 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:13.858 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:13.858 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:13.927 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:13.928 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:13.929 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:13.930 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:13.931 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:13.932 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:13.933 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:13.934 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:18.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:18.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:18.935 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:19.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:19.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:19.055 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:19.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:19.058 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:19.059 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:24.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:24.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:24.060 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:24.230 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:24.231 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:24.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:24.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:24.235 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:24.236 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:29.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:29.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:29.237 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:29.288 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:29.349 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:29.349 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:29.349 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:29.349 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:29.349 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:29.349 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:29.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:29.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:29.351 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:29.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:29.353 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:29.354 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:34.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:34.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:34.355 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:34.406 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:34.407 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:34.490 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:34.495 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:34.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:39.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:39.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:39.498 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:39.548 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:39.548 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:39.611 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:39.611 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:39.613 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:39.614 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:39.615 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:39.616 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:39.617 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:44.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:44.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:44.619 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:44.670 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:44.670 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:44.757 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:44.757 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:44.757 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:44.758 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:44.759 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:44.760 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:44.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:44.762 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:44.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:49.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:49.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:49.765 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:49.816 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:49.816 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:49.884 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:49.884 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:49.885 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:49.886 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:49.887 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:49.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:49.889 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:49.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T22:59:54.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:54.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T22:59:54.891 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T22:59:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T22:59:54.942 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T22:59:55.027 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T22:59:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:55.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:55.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:55.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T22:59:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T22:59:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T22:59:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T22:59:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T22:59:55.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T22:59:55.032 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T22:59:55.033 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:00.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:00.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:00.034 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:00.085 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:00.166 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:00.166 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:00.166 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:00.166 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:00.166 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:00.167 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:00.168 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:00.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:00.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:00.171 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:00.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:00.171 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:00.171 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:00.171 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:00.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:00.171 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:00.172 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:05.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:05.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:05.173 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:05.224 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:05.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:05.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:05.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:05.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:05.298 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:05.299 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:05.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:10.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:10.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:10.300 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:10.424 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:10.424 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:10.425 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:10.426 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:10.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:10.428 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:10.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:10.429 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:10.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:10.429 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:10.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:10.429 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:10.430 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:15.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:15.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:15.431 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:15.496 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:15.631 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:15.631 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:15.632 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:15.633 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:15.634 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:15.635 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:15.636 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:15.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:20.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:20.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:20.639 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:20.748 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:20.749 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:20.750 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:20.751 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:20.752 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:20.753 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:25.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:25.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:25.755 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:25.887 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:25.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:25.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:25.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:25.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:25.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:25.893 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:25.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:25.893 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:25.894 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:30.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:30.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:30.895 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:30.947 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:31.026 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:31.027 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:31.028 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:31.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:31.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:31.031 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:31.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:36.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:36.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:36.033 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:36.084 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:36.151 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:36.151 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:36.152 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:36.156 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:36.157 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:41.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:41.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:41.159 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:41.222 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:41.223 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:41.331 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:41.331 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:41.331 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:41.331 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:41.331 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:41.331 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:41.332 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:41.333 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:41.335 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:41.336 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:41.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:41.336 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:41.337 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:46.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2024-08-21T23:00:46.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:46.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:46.340 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:46.441 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:46.461 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:55:38.254894Z" 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:46.462 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T23:00:46.463 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T23:00:46.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:46.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:46.463 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:46.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:46.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:46.585 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:46.585 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:46.585 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:46.585 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:46.585 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:46.585 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:46.585 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:46.586 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:46.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:46.589 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:46.590 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:46.590 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:46.590 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:46.591 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:51.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:51.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:51.592 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:51.644 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:51.644 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:51.727 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:51.728 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:51.729 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:51.730 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:51.731 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:00:56.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:56.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:00:56.733 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:00:56.784 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:00:56.784 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:00:56.862 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:56.863 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:00:56.864 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:00:56.865 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:00:56.866 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:00:56.867 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:00:56.868 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:01.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:01.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:01.870 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:01.994 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:01.998 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:02.000 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:07.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:07.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:07.001 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:07.045 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:07.045 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:07.105 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:07.105 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:07.105 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:07.105 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:07.106 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:07.107 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:07.108 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:07.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:07.110 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:07.111 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:12.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:12.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:12.113 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:12.164 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:12.164 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:12.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:12.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:12.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:12.236 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:12.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:12.237 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:12.238 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:17.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:17.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:17.239 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:17.378 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:17.379 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:17.380 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:17.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:17.382 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:17.383 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:17.384 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:17.384 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:17.385 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:22.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:22.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:22.387 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:22.437 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:22.496 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:22.496 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:22.497 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:22.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:22.499 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:22.500 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:22.501 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:22.502 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:22.503 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:22.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:22.503 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:22.503 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:22.503 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:22.503 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:22.503 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:22.504 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:27.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:27.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:27.505 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:27.556 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:27.629 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:27.629 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:27.629 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:27.629 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:27.630 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:27.631 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:27.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:27.633 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:27.634 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:27.634 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:27.634 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:27.635 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:32.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:32.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:32.636 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:32.687 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:32.742 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:32.742 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:32.742 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:32.742 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:32.742 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:32.742 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:32.743 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:32.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:32.745 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:32.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:32.747 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:32.747 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:32.747 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:32.747 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:32.747 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:32.748 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:37.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:37.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:37.749 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:37.870 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:37.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:37.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:37.873 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:37.874 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:37.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:37.875 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:37.875 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:42.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:42.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:42.877 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:42.987 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:42.987 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:42.987 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:42.987 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:42.987 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:42.987 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:42.987 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:42.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:42.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:42.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:42.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:42.992 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:42.992 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:42.992 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:42.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:42.992 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:42.993 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:47.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:47.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:47.994 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:48.045 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:48.121 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:48.121 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:48.121 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:48.121 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:48.121 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:48.121 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:48.122 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:48.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:48.124 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:48.125 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:48.126 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:48.126 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:48.127 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:53.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:53.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:53.128 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:53.237 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:53.237 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:53.237 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:53.238 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:53.239 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:53.240 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:53.242 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:53.243 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:01:58.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:58.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:01:58.244 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:01:58.288 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:01:58.288 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:01:58.344 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:01:58.344 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:01:58.344 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:01:58.344 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:01:58.344 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:01:58.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:58.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:58.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:01:58.348 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:01:58.349 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:01:58.349 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:01:58.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:01:58.349 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:01:58.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:03.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:03.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:03.351 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:03.475 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:03.476 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:03.477 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:03.478 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:03.479 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:03.480 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:03.481 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:08.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:08.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:08.483 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:08.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:08.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:08.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:08.602 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:08.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:08.604 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:08.605 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:13.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:13.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:13.606 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:13.747 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:13.748 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:18.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:18.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:18.749 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:18.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:18.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:18.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:18.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:18.872 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:18.872 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:18.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:18.872 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:18.873 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:23.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:23.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:23.874 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:23.927 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:23.989 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:23.989 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:23.989 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:23.990 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:23.991 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:23.992 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:23.993 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:23.994 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:23.995 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:28.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:28.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:28.997 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:29.047 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:29.048 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:29.133 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:29.134 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:29.135 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:29.136 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:29.137 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:29.138 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:29.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:34.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:34.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:34.141 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:34.192 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:34.193 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:34.253 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:34.259 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:34.260 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:39.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:39.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:39.262 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:39.312 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:39.312 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:39.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:39.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:39.380 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:39.381 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:39.382 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:39.383 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:39.383 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:39.384 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:44.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:44.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:44.386 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:44.439 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:44.520 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:44.520 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:44.521 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:44.522 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:44.523 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:44.524 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:44.525 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:44.526 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:49.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:49.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:49.528 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:49.571 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:49.572 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:49.628 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:49.628 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:49.628 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:49.628 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:49.628 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:49.628 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:49.629 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:49.630 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:49.631 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:49.632 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:49.633 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:49.634 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:54.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:54.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:54.635 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:54.684 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:54.757 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:54.758 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:54.759 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:54.760 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:54.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:54.762 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:54.763 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:02:59.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:59.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:02:59.765 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:02:59.816 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:02:59.816 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:02:59.887 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:59.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:02:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:02:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:02:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:02:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:02:59.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:02:59.895 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:02:59.895 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:02:59.895 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:02:59.895 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:02:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:02:59.898 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:04.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:04.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:04.900 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:05.028 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:05.029 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:05.030 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:05.031 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:05.032 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:05.033 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:05.034 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:10.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:10.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:10.036 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:10.159 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:10.159 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:10.159 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:10.159 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:10.159 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:10.159 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:10.160 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:10.161 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:10.162 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:10.163 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:10.164 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:10.165 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:15.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:15.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:15.166 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:15.213 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:15.213 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:15.284 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:15.284 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:15.285 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:15.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:15.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:15.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:15.289 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:15.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:20.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:20.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:20.291 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:20.414 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:20.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:20.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:20.417 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:20.418 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:20.419 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:20.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:20.419 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:20.420 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:25.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:25.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:25.421 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:25.465 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:25.465 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:25.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:25.537 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:25.538 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:25.539 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:25.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:25.541 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:25.541 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:30.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:30.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:30.543 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:30.594 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:30.594 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:30.675 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:30.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:30.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:30.678 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:30.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:30.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:30.681 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:30.682 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:35.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:35.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:35.683 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:35.734 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:35.734 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:35.800 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:35.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:40.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:40.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:40.803 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:40.851 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:40.852 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:40.908 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:40.908 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:40.908 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:40.908 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:40.908 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:40.908 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:40.909 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:40.910 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:40.911 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:40.912 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:40.913 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:40.914 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:40.914 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:40.915 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:45.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:45.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:45.917 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:45.963 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:46.040 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:46.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:46.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:46.043 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:46.044 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:46.045 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:46.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:46.045 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:46.045 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:46.045 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:46.045 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:46.045 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:46.046 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:51.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:51.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:51.047 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:51.178 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:51.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:51.180 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:51.181 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:51.182 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:51.183 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:51.184 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:03:56.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:56.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:03:56.186 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:03:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:03:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:03:56.293 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:03:56.294 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:03:56.295 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:03:56.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:03:56.297 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:03:56.298 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:03:56.299 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:01.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:01.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:01.300 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:01.351 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:01.351 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:01.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:01.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:01.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:01.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:01.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:01.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:01.433 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:01.434 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:06.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:06.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:06.435 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:06.486 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:06.487 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:06.574 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:06.574 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:06.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:06.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:06.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:06.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:06.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:06.579 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:06.580 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:11.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:11.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:11.581 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:11.712 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:11.712 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:11.712 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:11.712 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:11.713 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:11.714 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:11.715 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:11.716 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:11.717 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:11.718 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:16.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:16.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:16.720 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:16.765 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:16.766 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:16.825 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:16.828 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:16.829 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:16.830 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:16.831 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:21.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:21.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:21.832 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:21.877 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:21.936 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:21.936 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:21.936 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:21.936 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:21.936 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:21.937 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:21.937 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:21.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:21.937 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:21.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:21.937 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:21.938 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:21.938 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:21.938 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:21.938 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:21.938 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:21.938 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:21.939 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:21.939 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:21.939 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:21.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:21.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:21.939 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:21.940 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:21.940 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:21.940 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:21.940 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:21.940 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:21.940 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:21.941 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:21.942 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:21.943 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:21.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:21.944 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:21.945 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:26.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:26.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:26.947 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:26.991 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:26.991 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:27.059 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:27.061 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:27.063 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:27.064 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:27.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:32.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:32.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:32.066 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:32.181 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:32.181 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:32.181 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:32.182 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:32.183 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:32.184 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:32.185 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:32.186 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:32.187 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:37.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:37.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:37.189 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:37.240 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:37.241 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:37.309 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:37.309 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:37.309 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:37.309 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:37.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:37.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:37.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:37.313 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:37.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:37.314 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:37.314 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:37.314 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:37.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:37.314 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:37.315 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:42.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:42.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:42.316 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:42.440 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:42.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:42.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:42.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:42.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:42.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:42.446 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:42.446 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:47.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:47.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:47.448 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:47.579 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:47.579 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:47.580 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:47.582 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:47.583 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:47.584 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:47.585 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:52.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:52.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:52.587 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:52.631 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:52.631 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:52.701 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:52.701 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:52.701 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:52.701 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:52.701 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:52.702 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:52.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:52.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:52.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:52.706 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:52.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:52.706 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:52.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:04:57.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:57.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:04:57.708 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:04:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:04:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:04:57.830 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:04:57.830 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:04:57.830 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:04:57.830 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:04:57.830 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:04:57.830 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:04:57.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:57.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:57.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:04:57.834 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:04:57.835 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:04:57.835 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:04:57.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:04:57.835 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:04:57.836 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:02.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:02.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:02.838 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:02.888 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:02.888 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:02.952 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:02.958 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:02.959 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:07.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:07.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:07.960 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:08.078 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:08.078 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:08.080 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:08.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:08.082 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:08.083 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:08.084 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:13.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:13.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:13.085 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:13.136 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:13.137 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:13.220 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:13.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:13.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:13.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:13.224 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:13.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:13.224 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:13.224 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:13.224 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:13.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:13.225 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:13.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:18.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:18.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:18.227 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:18.335 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:18.335 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:18.335 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:18.335 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:18.335 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:18.335 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:18.335 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:18.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:18.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:18.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:18.339 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:18.340 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:18.340 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:18.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:18.340 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:18.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:23.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:23.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:23.342 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:23.388 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:23.459 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:23.459 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:23.459 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:23.459 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:23.459 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:23.460 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:23.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:23.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:23.463 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:23.464 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:23.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:23.464 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:23.465 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:28.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:28.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:28.466 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:28.517 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:28.603 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:28.603 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:28.603 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:28.603 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:28.604 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:28.605 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:28.606 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:28.607 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:28.608 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:28.609 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:33.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:33.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:33.611 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:33.665 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:33.666 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:33.729 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:33.730 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:33.731 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:33.732 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:33.733 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:33.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:33.735 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:33.736 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:38.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:38.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:38.737 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:38.789 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:38.789 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:38.849 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:38.849 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:38.849 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:38.849 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:38.849 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:38.849 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:38.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:38.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:38.853 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:38.855 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:43.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:43.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:43.857 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:43.907 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:43.983 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:43.985 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:43.986 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:43.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:43.988 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:43.988 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:48.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:48.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:05:48.991 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:05:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:49.044 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:49.120 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:49.120 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:49.121 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:05:49.123 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:49.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:49.125 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:49.126 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:05:54.128 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2024-08-21T23:05:54.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.129 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T23:05:54.173 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:54.174 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2fdf1079-7599-477a-a848-6b8b3507e9d9", 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "remove_zg", 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "remove_zg", 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "false", 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [], 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T23:05:54.202 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "", 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [], 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [], 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "", 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.203 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:54.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.204 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T23:05:54.246 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:54.246 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "73210daf-7650-49d6-908d-488c7f894c4c", 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "remove", 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "domain_root": "remove.rgw.meta:root", 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "control_pool": "remove.rgw.control", 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "gc_pool": "remove.rgw.log:gc", 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "lc_pool": "remove.rgw.log:lc", 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "log_pool": "remove.rgw.log", 2024-08-21T23:05:54.274 INFO:teuthology.orchestra.run.smithi116.stdout: "intent_log_pool": "remove.rgw.log:intent", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "usage_log_pool": "remove.rgw.log:usage", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "roles_pool": "remove.rgw.meta:roles", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "reshard_pool": "remove.rgw.log:reshard", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "otp_pool": "remove.rgw.otp", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "system_key": { 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "access_key": "", 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: "secret_key": "" 2024-08-21T23:05:54.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_pools": [ 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "default-placement", 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "index_pool": "remove.rgw.buckets.index", 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": { 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD": { 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "data_pool": "remove.rgw.buckets.data" 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "index_type": 0, 2024-08-21T23:05:54.276 INFO:teuthology.orchestra.run.smithi116.stdout: "inline_data": "true" 2024-08-21T23:05:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:54.277 INFO:teuthology.orchestra.run.smithi116.stdout: "notif_pool": "remove.rgw.log:notif" 2024-08-21T23:05:54.277 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:54.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.279 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T23:05:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:54.437 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 6, 2024-08-21T23:05:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "predecessor_uuid": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T23:05:54.437 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": [], 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "period_map": { 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "zonegroups": [ 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2fdf1079-7599-477a-a848-6b8b3507e9d9", 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "remove_zg", 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "remove_zg", 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "false", 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [], 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T23:05:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "73210daf-7650-49d6-908d-488c7f894c4c", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "73210daf-7650-49d6-908d-488c7f894c4c", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "remove", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [], 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "false", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:54.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T23:05:54.440 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a", 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:54.441 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a2", 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8001" 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "false", 2024-08-21T23:05:54.442 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a1", 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.443 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T23:05:54.444 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T23:05:54.445 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T23:05:54.445 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:54.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.445 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.445 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T23:05:54.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:54.445 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "b", 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "false", 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:54.446 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b2", 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8001" 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.447 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T23:05:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b1", 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.449 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T23:05:54.450 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "short_zone_ids": [ 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 1441323143 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 4021227367 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 943360455 2024-08-21T23:05:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "73210daf-7650-49d6-908d-488c7f894c4c", 2024-08-21T23:05:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 2516360235 2024-08-21T23:05:54.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T23:05:54.453 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 3475667924 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zonegroup": "a", 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "period_config": { 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T23:05:54.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.455 INFO:teuthology.orchestra.run.smithi116.stdout: "user_ratelimit": { 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_ratelimit": { 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T23:05:54.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "anonymous_ratelimit": { 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T23:05:54.457 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:54.458 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_name": "test-realm", 2024-08-21T23:05:54.459 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:54.459 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:54.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.460 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T23:05:54.545 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:54.545 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:54.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T23:05:54.579+0000 7fb03be1d7c0 0 zone id 73210daf-7650-49d6-908d-488c7f894c4c is not a part of zonegroup b 2024-08-21T23:05:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T23:05:54.581+0000 7fb03be1d7c0 0 zone id 73210daf-7650-49d6-908d-488c7f894c4c is not a part of zonegroup a 2024-08-21T23:05:54.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T23:05:54.581+0000 7fb03be1d7c0 0 zone id 73210daf-7650-49d6-908d-488c7f894c4c is not a part of zonegroup default 2024-08-21T23:05:54.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.595 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T23:05:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:54.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T23:05:54.686+0000 7f0ea3d467c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2024-08-21T23:05:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-21T23:05:54.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.693 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T23:05:54.739 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:54.739 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:54.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2024-08-21T23:05:54.794 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2024-08-21T23:05:54.854 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:05:54.854 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 7, 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "predecessor_uuid": "c9075436-66f3-4d8c-a9e4-e295f6fc0789", 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": [], 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "period_map": { 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "zonegroups": [ 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.004 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "a", 2024-08-21T23:05:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a", 2024-08-21T23:05:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T23:05:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "true", 2024-08-21T23:05:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T23:05:55.005 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T23:05:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T23:05:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T23:05:55.006 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T23:05:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a2", 2024-08-21T23:05:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8001" 2024-08-21T23:05:55.007 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:55.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "false", 2024-08-21T23:05:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.009 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "a1", 2024-08-21T23:05:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi116.front.sepia.ceph.com:8000" 2024-08-21T23:05:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.010 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T23:05:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T23:05:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.012 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T23:05:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T23:05:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T23:05:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T23:05:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:55.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T23:05:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T23:05:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T23:05:55.014 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "b", 2024-08-21T23:05:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b", 2024-08-21T23:05:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "api_name": "", 2024-08-21T23:05:55.015 INFO:teuthology.orchestra.run.smithi116.stdout: "is_master": "false", 2024-08-21T23:05:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T23:05:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames": [], 2024-08-21T23:05:55.016 INFO:teuthology.orchestra.run.smithi116.stdout: "hostnames_s3website": [], 2024-08-21T23:05:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T23:05:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "zones": [ 2024-08-21T23:05:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T23:05:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b2", 2024-08-21T23:05:55.017 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8001" 2024-08-21T23:05:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:55.018 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:55.019 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T23:05:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "id": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T23:05:55.020 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "b1", 2024-08-21T23:05:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "endpoints": [ 2024-08-21T23:05:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "http://smithi178.front.sepia.ceph.com:8000" 2024-08-21T23:05:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "log_meta": "false", 2024-08-21T23:05:55.021 INFO:teuthology.orchestra.run.smithi116.stdout: "log_data": "true", 2024-08-21T23:05:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_index_max_shards": 11, 2024-08-21T23:05:55.022 INFO:teuthology.orchestra.run.smithi116.stdout: "read_only": "false", 2024-08-21T23:05:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "tier_type": "", 2024-08-21T23:05:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from_all": "true", 2024-08-21T23:05:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_from": [], 2024-08-21T23:05:55.053 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_zone": "" 2024-08-21T23:05:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "placement_targets": [ 2024-08-21T23:05:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default-placement", 2024-08-21T23:05:55.054 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-08-21T23:05:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_classes": [ 2024-08-21T23:05:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "STANDARD" 2024-08-21T23:05:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.055 INFO:teuthology.orchestra.run.smithi116.stdout: "default_placement": "default-placement", 2024-08-21T23:05:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_policy": { 2024-08-21T23:05:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: "groups": [] 2024-08-21T23:05:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.056 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-08-21T23:05:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "short_zone_ids": [ 2024-08-21T23:05:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "11250536-b0eb-44b2-8efe-f0ea767669ff", 2024-08-21T23:05:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 1441323143 2024-08-21T23:05:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.057 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "3ccc7c5a-b8fe-4a58-84f4-d904ee91d189", 2024-08-21T23:05:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 4021227367 2024-08-21T23:05:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 943360455 2024-08-21T23:05:55.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:05:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "fe48b235-c997-4b77-a96a-2d49ab4ea593", 2024-08-21T23:05:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 3475667924 2024-08-21T23:05:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:05:55.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zonegroup": "a", 2024-08-21T23:05:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "master_zone": "5407971e-5f7f-4ba8-8316-03f497e31665", 2024-08-21T23:05:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "period_config": { 2024-08-21T23:05:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_quota": { 2024-08-21T23:05:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T23:05:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T23:05:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T23:05:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T23:05:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T23:05:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "user_quota": { 2024-08-21T23:05:55.061 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false, 2024-08-21T23:05:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "check_on_raw": false, 2024-08-21T23:05:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size": -1, 2024-08-21T23:05:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max_size_kb": 0, 2024-08-21T23:05:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max_objects": -1 2024-08-21T23:05:55.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.065 INFO:teuthology.orchestra.run.smithi116.stdout: "user_ratelimit": { 2024-08-21T23:05:55.065 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T23:05:55.066 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T23:05:55.066 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T23:05:55.066 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T23:05:55.066 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T23:05:55.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.066 INFO:teuthology.orchestra.run.smithi116.stdout: "bucket_ratelimit": { 2024-08-21T23:05:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T23:05:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T23:05:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T23:05:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T23:05:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T23:05:55.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "anonymous_ratelimit": { 2024-08-21T23:05:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_ops": 0, 2024-08-21T23:05:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_ops": 0, 2024-08-21T23:05:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max_read_bytes": 0, 2024-08-21T23:05:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max_write_bytes": 0, 2024-08-21T23:05:55.068 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": false 2024-08-21T23:05:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:05:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:05:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_id": "c5b47b13-9a2b-401b-92e0-5cba73a5c237", 2024-08-21T23:05:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_name": "test-realm", 2024-08-21T23:05:55.069 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:05:55.069 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:05:55.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2024-08-21T23:05:55.072 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-36 2024-08-21T23:05:55.801 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-37 2024-08-21T23:05:55.867 INFO:rgw_multi.tests:create bucket zone=b1 name=qlihov-38 2024-08-21T23:05:59.249 INFO:rgw_multi.tests:create bucket zone=b2 name=qlihov-39 2024-08-21T23:05:59.524 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-40 2024-08-21T23:05:59.548 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-41 2024-08-21T23:05:59.584 INFO:rgw_multi.tests:create bucket zone=b1 name=qlihov-42 2024-08-21T23:05:59.890 INFO:rgw_multi.tests:create bucket zone=b2 name=qlihov-43 2024-08-21T23:06:02.977 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ok 2024-08-21T23:06:02.977 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-44 2024-08-21T23:06:03.000 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-45 2024-08-21T23:06:03.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:03.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:03.053 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:03.097 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:03.152 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T23:06:03.176 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T23:06:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T23:06:03.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281563.033560_389.1", 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T23:06:03.033560Z" 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:03.177 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T23:06:03.178 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T23:06:03.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:03.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:03.179 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:03.263 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:03.345 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:03.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:03.347 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:03.348 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:03.349 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:03.350 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:08.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:08.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:08.352 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:08.403 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:08.483 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:08.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:08.485 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:08.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:08.487 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:08.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:08.488 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:08.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:08.488 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:08.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:13.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:13.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:13.494 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:13.544 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:13.622 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:13.623 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:13.623 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:13.623 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:13.623 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:13.623 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:13.623 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:13.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:13.624 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:13.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:13.624 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:13.624 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:13.624 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:13.624 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:13.625 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:13.625 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:13.625 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:13.625 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:13.625 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:13.625 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:13.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:13.626 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:13.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:13.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:13.629 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:13.629 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:13.629 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:13.629 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:13.629 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:13.630 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:18.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:18.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:18.632 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:18.747 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:18.747 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:18.747 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:18.747 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:18.748 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:18.749 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:18.750 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:18.751 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:18.752 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:18.753 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:23.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:23.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:23.755 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:23.805 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:23.806 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:23.883 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:23.884 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:23.885 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:23.886 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:23.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:23.888 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:23.889 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:23.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:28.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:28.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:28.892 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:28.943 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:28.944 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:29.031 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:29.031 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:29.032 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:29.033 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:29.035 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:29.036 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:29.037 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:34.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:34.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:34.039 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:34.089 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:34.090 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:34.169 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:34.174 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:34.174 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:39.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:39.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:39.176 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:39.297 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:39.298 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:39.299 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:39.300 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:39.301 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:39.302 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:39.303 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:39.304 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:44.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:44.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:44.305 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:44.356 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:44.356 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:44.441 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:44.441 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:44.441 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:44.441 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:44.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:44.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:44.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:44.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:44.446 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:44.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:49.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:49.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:49.449 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:49.499 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:49.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:49.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:49.569 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:49.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:49.571 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:49.572 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:49.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:49.572 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:49.573 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:54.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:54.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:54.574 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:54.624 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:54.695 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:54.696 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:54.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:54.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:54.699 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:54.700 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:54.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:54.701 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:54.702 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:06:59.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:59.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:06:59.703 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:06:59.754 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:06:59.754 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:06:59.837 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:06:59.837 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:06:59.837 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:06:59.837 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:06:59.837 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:06:59.837 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:59.838 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:06:59.839 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:59.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:59.841 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:06:59.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:06:59.842 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:06:59.842 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:06:59.842 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:06:59.842 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:06:59.842 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:06:59.843 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:04.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:04.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:04.845 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:04.895 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:04.896 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:04.976 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:04.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:04.976 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:04.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:04.978 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:04.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:04.980 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:04.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:04.981 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:04.981 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:04.981 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:04.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:04.981 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:04.982 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:09.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:09.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:09.986 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:10.036 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:10.036 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:10.117 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:10.118 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:10.119 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:10.120 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:10.121 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:10.121 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:10.122 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:15.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:15.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:15.123 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:15.174 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:15.256 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:15.257 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:15.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:15.259 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:15.260 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:15.261 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:15.262 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:20.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:20.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:20.264 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:20.315 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:20.315 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:20.399 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:20.400 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:20.401 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:20.402 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:20.403 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:20.404 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:20.405 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:25.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:25.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:25.407 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:25.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:30.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:30.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:30.538 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:30.662 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:30.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:30.664 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:30.665 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:30.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:30.667 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:30.668 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:35.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:35.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:35.669 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:35.720 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:35.720 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:35.806 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:35.807 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:40.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:40.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:40.809 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:40.860 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:40.861 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:40.947 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:40.947 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:40.947 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:40.947 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:40.947 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:40.948 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:40.949 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:40.950 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:40.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:40.950 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:40.950 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:40.950 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:40.950 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:40.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:40.951 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:40.951 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724280938.254894_297.1", 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:55:38.254894Z", 2024-08-21T23:07:40.952 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:40.953 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:40.954 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724280938.254894_297.1 2024-08-21T23:07:45.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:45.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:45.956 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:46.007 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:46.008 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:46.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:46.082 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:46.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:46.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:46.086 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:46.086 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:46.086 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:46.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:46.086 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:46.087 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:07:51.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:51.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:51.089 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:51.139 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:51.214 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:51.214 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:51.214 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:51.214 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:51.214 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:51.215 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:51.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:51.217 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281555.758356_326.1", 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:55.758356Z", 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:51.218 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:51.219 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:51.219 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:51.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:51.219 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:51.220 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281555.758356_326.1 2024-08-21T23:07:56.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:56.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:07:56.221 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:07:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:07:56.270 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:07:56.352 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:07:56.352 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:07:56.352 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:07:56.352 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:07:56.352 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:07:56.352 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:56.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:56.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:07:56.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:07:56.356 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:07:56.357 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:07:56.357 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:07:56.357 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:07:56.357 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:07:56.358 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:01.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:01.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:01.359 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:01.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:01.411 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:01.483 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:01.483 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:01.483 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:01.484 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:01.485 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:01.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:01.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:01.488 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:01.489 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:08:06.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:06.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:06.491 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:06.541 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:06.541 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:06.605 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:06.606 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:06.607 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:06.609 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:06.610 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:11.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:11.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:11.612 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:11.663 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:11.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:11.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:11.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:11.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:11.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:11.727 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:11.727 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:11.727 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:11.727 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:11.727 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:11.728 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:16.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:16.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:16.729 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:16.780 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:16.780 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:16.841 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:16.841 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:16.841 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:16.841 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:16.841 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:16.844 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:16.845 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:16.846 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:16.847 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:21.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:21.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:21.848 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:21.894 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:21.894 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:21.971 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:21.971 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:21.972 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:21.972 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:21.972 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:21.972 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:21.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:21.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:21.973 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:21.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:21.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:21.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:21.973 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:21.973 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:21.973 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:21.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:21.974 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:21.974 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:21.974 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:21.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:21.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:21.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:21.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:21.975 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:21.975 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:21.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:21.981 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:08:26.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:26.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:26.983 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:27.090 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:27.091 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:27.091 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:27.091 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:27.091 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:27.091 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:27.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:27.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:27.092 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:27.093 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:27.094 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:27.095 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:27.095 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:27.095 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:27.095 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:27.095 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:27.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:27.096 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:08:27.097 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:27.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:27.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:27.097 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:27.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:27.097 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:27.098 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:27.099 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:08:32.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:32.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:32.100 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:32.231 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:32.232 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:32.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:32.235 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:32.237 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:37.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:37.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:37.239 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:37.351 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:37.351 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:37.351 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:37.351 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:37.351 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:37.351 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:37.351 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:37.352 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:37.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:37.355 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:37.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:37.356 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:37.356 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:37.356 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:37.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:37.356 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:37.357 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:42.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:42.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:42.358 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:42.409 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:42.489 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:42.489 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:42.489 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:42.489 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:42.489 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:42.490 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:42.491 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:42.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:42.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:42.494 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:42.495 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:47.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:47.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:47.497 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:47.542 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:47.542 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:47.614 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:47.615 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:47.616 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:47.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:47.618 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:47.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:47.619 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:47.619 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:47.619 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:47.619 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:47.619 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:47.620 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:52.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:52.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:52.622 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:52.679 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:52.790 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:52.790 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:52.790 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:52.790 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:52.790 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:52.790 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:52.790 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:52.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:52.792 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:52.793 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:52.794 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:52.795 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:52.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:08:57.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:57.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:08:57.797 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:08:57.846 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:08:57.846 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:08:57.944 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:08:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:08:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:08:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:08:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:08:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:08:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:08:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:08:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:08:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:08:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:08:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:08:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:08:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:08:57.950 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:57.950 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:57.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:57.950 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:57.951 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:08:57.951 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:57.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:57.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:08:57.951 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:57.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:57.952 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:57.952 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:08:57.952 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:57.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:57.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:57.953 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:08:57.953 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:08:57.953 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:08:57.953 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:08:57.953 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:08:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:08:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:08:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:08:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:08:57.954 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:08:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:08:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:08:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:08:57.955 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:08:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:08:57.956 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:08:57.956 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:08:57.958 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:02.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:02.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:02.959 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:03.004 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:03.004 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:03.109 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:03.110 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:03.111 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:03.112 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:03.113 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:03.114 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:03.115 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:03.116 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:08.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:08.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:08.117 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:08.161 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:08.161 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:08.223 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:08.223 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:08.223 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:08.228 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:08.229 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:13.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:13.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:13.231 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:13.341 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:13.341 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:13.341 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:13.342 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:13.343 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:13.344 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:13.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:13.346 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:13.347 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:09:18.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:18.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:18.349 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:18.399 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:18.465 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:18.465 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:18.465 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:18.465 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:18.465 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:18.465 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:18.466 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:18.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:18.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:18.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:18.470 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:18.470 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:18.470 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:18.470 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:18.470 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:18.471 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:09:23.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:23.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:23.473 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:23.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:23.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:23.584 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:23.585 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:23.586 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:23.587 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:23.587 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:23.588 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:28.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:28.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:28.590 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:28.642 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:28.643 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:28.721 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:28.721 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:28.721 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:28.722 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:28.723 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:28.724 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:28.725 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:28.726 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:28.727 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:33.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:33.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:33.729 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:33.848 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:33.848 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:33.848 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:33.848 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:33.848 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:33.848 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:33.848 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:33.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:33.850 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:33.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:33.852 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:33.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:33.853 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:33.854 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:38.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:38.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:38.856 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:38.969 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:38.969 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:38.969 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:38.969 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:38.970 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:38.971 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:38.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:38.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:38.974 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:38.975 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:43.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:43.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:43.977 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:44.027 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:44.028 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:44.110 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:44.111 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:49.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:49.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:49.112 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:49.160 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:49.220 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:49.220 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:49.220 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:49.221 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:49.222 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:49.223 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:49.224 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:49.225 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:49.226 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:09:54.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:54.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:54.227 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:54.351 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:54.352 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:54.353 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:54.354 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:54.356 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:54.357 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:09:59.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:59.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:09:59.359 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:09:59.406 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:09:59.406 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:09:59.485 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:09:59.485 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:59.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:59.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:59.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.559752_359.1", 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:09:59.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.559752Z", 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:09:59.490 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:09:59.491 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.559752_359.1 2024-08-21T23:10:04.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:04.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:04.493 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:04.544 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:04.545 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:04.614 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:04.614 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:04.615 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:04.616 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:04.617 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.559752_359.1", 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.559752Z", 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:04.619 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:04.620 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.559752_359.1 2024-08-21T23:10:09.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:09.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:09.622 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:09.667 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:09.668 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:09.733 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:09.733 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:09.733 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:09.733 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:09.733 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:09.733 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:09.734 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:09.735 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:09.736 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.178070_349.1", 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.178070Z", 2024-08-21T23:10:09.737 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:09.738 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:09.739 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.178070_349.1 2024-08-21T23:10:14.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:14.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:14.741 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:14.792 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:14.793 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:14.874 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:14.875 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:14.876 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:14.877 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:14.878 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:14.879 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:14.879 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:14.880 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:10:19.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:19.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:19.881 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:19.929 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:19.930 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:19.994 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:19.994 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:19.994 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:19.994 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:19.994 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:19.994 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:19.995 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:19.996 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:19.997 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:19.998 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:19.999 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:20.000 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:10:25.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:25.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:25.002 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:25.053 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:25.053 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:25.129 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:25.130 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:25.131 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:25.132 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:25.133 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:25.134 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:25.135 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:25.135 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:25.135 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:25.136 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:10:30.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:30.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:30.138 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:30.190 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:30.191 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:30.268 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:30.269 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:30.270 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:30.271 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:30.272 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:30.272 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:30.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:30.272 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:30.273 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:10:35.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:35.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:35.275 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:35.326 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:35.326 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:35.414 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:35.419 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:35.420 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:10:40.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:40.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:40.422 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:40.472 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:40.472 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:40.535 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:40.535 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:40.535 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:40.536 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:40.537 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:40.538 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:40.539 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:40.540 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:40.541 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:10:45.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:45.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:45.543 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:45.594 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:45.689 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:45.690 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:45.691 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:45.692 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:45.693 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:45.694 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:45.695 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.167821_344.1 2024-08-21T23:10:50.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:50.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:50.697 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:50.748 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:50.748 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:50.826 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:50.826 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:50.826 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:50.827 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:50.828 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:50.829 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:50.830 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:10:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:50.831 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:50.832 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:50.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:50.832 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:50.833 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:10:55.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:55.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:10:55.834 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:10:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:10:55.879 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:10:55.941 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:55.942 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:55.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:55.944 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:10:55.945 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:10:55.946 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:10:55.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:10:55.947 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:10:55.948 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:11:00.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:00.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:00.949 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:00.994 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:00.994 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:01.073 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:01.073 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:01.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:01.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:01.076 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:01.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:01.078 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:01.078 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:01.078 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:01.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:01.078 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:01.079 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:11:06.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:06.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:06.080 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:06.131 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:06.189 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:06.189 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:06.190 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:06.191 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:06.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:06.193 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:06.194 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:06.195 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:06.196 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281563.033560_389.1 target=1_1724281559.428644_350.1 2024-08-21T23:11:11.197 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... FAIL 2024-08-21T23:11:11.198 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-46 2024-08-21T23:11:11.226 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-47 2024-08-21T23:11:11.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:11.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:11.458 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:11.502 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:11.502 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T23:11:11.569 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T23:11:11.570 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.570 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:11.570 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T23:11:11.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.571 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:11.571 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T23:11:11.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.571 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.571 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T23:11:11.572 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T23:11:11.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.572 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:11:11.572 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T23:11:11.246527Z" 2024-08-21T23:11:11.572 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:11.573 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T23:11:11.574 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T23:11:11.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:11.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:11.575 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.699 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.701 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:11.702 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:11.703 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:11.704 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:11.704 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:11.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:11.704 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:11.706 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.428644_350.1 2024-08-21T23:11:16.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:16.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:16.708 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:16.758 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:16.759 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:16.832 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:16.832 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:16.832 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:16.832 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:16.832 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:16.832 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:16.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:16.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:16.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:16.836 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:16.837 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:16.837 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:16.837 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:16.838 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:21.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:21.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:21.839 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:21.899 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:22.012 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:22.012 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:22.012 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:22.012 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:22.012 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:22.013 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:22.014 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:22.015 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:22.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:22.017 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:22.018 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:27.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:27.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:27.019 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:27.063 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:27.063 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:27.137 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:27.137 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:27.137 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:27.138 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:27.139 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:27.140 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:27.141 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:27.142 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:27.143 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:32.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:32.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:32.145 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:32.197 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:32.197 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:32.277 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:32.278 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:32.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:32.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:32.279 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:32.279 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:32.279 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:32.280 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:32.281 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:32.282 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:32.283 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:37.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:37.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:37.285 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:37.337 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:37.414 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:37.415 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:37.416 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:37.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:37.418 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:37.419 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:37.420 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:37.421 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:42.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:42.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:42.423 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:42.494 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:42.494 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:42.598 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:42.598 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:42.598 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:42.598 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:42.599 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:42.600 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:42.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:42.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:42.603 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:42.604 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:47.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:47.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:47.605 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:47.716 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:47.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:47.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:47.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:47.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:47.721 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:47.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:47.721 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:47.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:47.721 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:47.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:47.721 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:47.722 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:52.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:52.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:52.723 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:52.768 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:52.768 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:52.818 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:52.818 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:52.819 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:52.820 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:52.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:52.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:52.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:52.824 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:52.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:11:57.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:57.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:11:57.826 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:11:57.881 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:11:57.881 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:11:57.942 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:11:57.942 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:11:57.942 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:11:57.942 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:57.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:11:57.944 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:11:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:57.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:11:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:57.946 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:11:57.947 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:11:57.948 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:11:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:11:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:11:57.949 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:11:57.949 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:11:57.950 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:02.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:02.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:02.952 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:03.072 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:03.073 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:03.074 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:03.075 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:03.076 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:03.077 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:03.078 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:03.080 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:08.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:08.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:08.082 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:08.129 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:08.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:08.213 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:13.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:13.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:13.215 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:13.265 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:13.265 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:13.324 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:13.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:13.326 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:13.327 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.178070_349.1", 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.178070Z", 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:13.328 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:13.329 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:13.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:13.329 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:13.330 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.178070_349.1 2024-08-21T23:12:18.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:18.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:18.331 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:18.375 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:18.375 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:18.429 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:18.429 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:18.429 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:18.429 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:18.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:18.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:18.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.178070_349.1", 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.178070Z", 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:18.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:18.434 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:18.435 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.178070_349.1 2024-08-21T23:12:23.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:23.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:23.436 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:23.549 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:23.550 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:23.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:23.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:23.552 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:23.552 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:23.552 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:23.552 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:23.552 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:23.552 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:23.553 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:23.553 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.905168_368.1", 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:23.554 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.905168Z", 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:23.555 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:23.556 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.905168_368.1 2024-08-21T23:12:28.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:28.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:28.558 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:28.606 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:28.659 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:28.660 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:28.661 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:28.662 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:28.663 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:28.664 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:28.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:28.664 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:28.664 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:28.664 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:28.664 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:28.664 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:28.665 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:33.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:33.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:33.666 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:33.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:33.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:33.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:33.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:33.809 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:33.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:33.810 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:33.810 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:33.810 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:33.810 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:33.810 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:33.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:38.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:38.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:38.812 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:38.863 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:38.864 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:38.947 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:38.948 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:38.949 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:38.950 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:38.951 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:38.952 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:38.953 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:43.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:43.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:43.954 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:43.999 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:43.999 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:44.063 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:44.068 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:44.069 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:49.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:49.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:49.071 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:49.117 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:49.117 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:49.171 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:49.172 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:49.173 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:49.174 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.559752_359.1", 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.559752Z", 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:49.175 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:49.177 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.559752_359.1 2024-08-21T23:12:54.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:54.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:54.179 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:54.284 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:54.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:54.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:54.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:54.289 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:54.289 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:12:59.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:59.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:12:59.291 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:12:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:12:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:12:59.425 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:59.426 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:12:59.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:59.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:12:59.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:12:59.430 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:12:59.431 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:04.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:04.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:04.434 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:04.556 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:04.557 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:04.558 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:04.559 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:04.560 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:04.561 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:04.562 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:09.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:09.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:09.563 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:09.675 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:09.675 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:09.675 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:09.675 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:09.676 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:09.677 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:09.678 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:09.679 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:09.680 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:09.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:14.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:14.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:14.682 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:14.733 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:14.733 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:14.813 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:14.814 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:14.815 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:14.816 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:14.817 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:14.818 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:14.819 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:14.819 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:14.820 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:19.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:19.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:19.822 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:19.872 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:19.872 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:19.930 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:19.931 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:19.931 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:19.931 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:19.931 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:19.931 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:19.932 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:19.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:19.932 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:19.932 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:19.933 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:19.933 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:19.933 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:19.933 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:19.933 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:19.933 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:19.934 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:19.934 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:19.934 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:19.934 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:19.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:19.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:19.935 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:19.935 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:19.935 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:19.935 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:19.936 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:19.936 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:19.936 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:19.936 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:19.936 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:19.937 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:19.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:19.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:19.937 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:19.937 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:19.938 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:19.938 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:19.938 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:19.938 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:19.938 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:19.939 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:19.939 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:19.939 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:19.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:19.939 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:19.940 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:19.940 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:19.940 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:19.940 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:19.940 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:19.940 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:19.941 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:19.941 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:19.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:19.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:19.942 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:19.942 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:19.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:19.942 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:19.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:19.943 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:19.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:19.943 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:19.945 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:24.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:24.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:24.947 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:24.997 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:25.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:25.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:25.065 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:25.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:25.067 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:25.067 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:25.067 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:25.067 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:25.067 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:25.068 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.428644_350.1 2024-08-21T23:13:30.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:30.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:30.070 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:30.176 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:30.180 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:30.181 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:30.182 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:35.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:35.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:35.184 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:35.235 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:35.235 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:35.317 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:35.322 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:35.323 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:40.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:40.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:40.325 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:40.374 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:40.374 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:40.449 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:40.450 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:40.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:40.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:40.453 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:40.454 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:40.455 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:40.456 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:45.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:45.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:45.457 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:45.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:45.507 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:45.564 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:45.564 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:45.564 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:45.564 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:45.564 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:45.564 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:45.565 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:45.566 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:45.567 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:45.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:45.569 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:45.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.428644_350.1 2024-08-21T23:13:50.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:50.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:50.571 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:50.623 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:50.684 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:50.684 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:50.684 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:50.686 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:50.687 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:50.688 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:50.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:50.690 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:50.691 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:50.692 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:50.693 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:13:55.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:55.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:13:55.695 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:13:55.746 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:13:55.746 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:13:55.806 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:13:55.806 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:13:55.806 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:55.807 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:55.808 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:55.809 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:13:55.810 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:13:55.811 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:13:55.812 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.428644_350.1 2024-08-21T23:14:00.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:00.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:00.814 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:00.859 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:00.859 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:00.941 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:00.941 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:00.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:00.945 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:00.946 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:00.946 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:00.947 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.428644_350.1 2024-08-21T23:14:05.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:05.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:05.949 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:05.994 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:05.994 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:06.082 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:06.083 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:06.084 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:06.085 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:06.087 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:06.088 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:06.088 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:06.089 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:11.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:11.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:11.090 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:11.142 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:11.142 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:11.212 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:11.212 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:11.212 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:11.212 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:11.212 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:11.212 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:11.213 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:11.214 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:11.215 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:11.216 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.428644_350.1", 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.428644Z", 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:11.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:11.218 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:11.219 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.428644_350.1 2024-08-21T23:14:16.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:16.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:16.220 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:16.271 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:16.271 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:16.342 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:16.342 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:16.343 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:16.344 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:16.345 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:16.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:16.347 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:16.348 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:21.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:21.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:21.349 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:21.400 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:21.400 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:21.464 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:21.465 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:21.466 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:21.467 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:21.468 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:21.469 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:21.470 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:26.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:26.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:26.472 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:26.593 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:26.594 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:26.595 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:26.596 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:26.597 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:26.598 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:26.599 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:31.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:31.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:31.601 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:31.646 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:31.704 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:31.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:31.706 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:31.707 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:31.708 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:31.709 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:31.710 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:31.710 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:31.711 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:36.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:36.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:36.712 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:36.755 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:36.841 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:41.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:41.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:41.843 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:41.894 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:41.972 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:41.972 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:41.973 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:41.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:41.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:41.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:41.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:41.977 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:41.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:46.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:46.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:46.980 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:47.030 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:47.031 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:47.088 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:47.093 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:47.094 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:47.094 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:47.094 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:47.094 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:47.095 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:52.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:52.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:52.097 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:52.212 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:52.213 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:52.214 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:52.215 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281559.167821_344.1", 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:52.216 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:05:59.167821Z", 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:52.217 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:52.218 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1, shard=3 master=1_1724281871.246527_404.1 target=1_1724281559.167821_344.1 2024-08-21T23:14:57.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:57.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:14:57.219 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:14:57.270 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:14:57.270 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:14:57.344 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:57.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:14:57.347 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:57.348 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:14:57.349 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:14:57.350 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:14:57.352 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:02.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:02.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:02.353 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:02.461 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:02.466 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:02.467 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:07.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:07.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:07.468 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:07.519 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:07.519 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:07.577 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:07.577 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:07.577 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:07.577 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:07.577 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:07.577 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:07.578 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:07.579 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:07.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:07.582 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:07.583 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:12.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:12.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:12.585 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:12.736 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:12.737 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:12.738 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:12.739 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:12.740 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:12.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:12.741 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:12.741 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:12.741 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:12.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:12.741 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:12.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:17.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:17.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:17.744 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:17.794 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:17.795 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:17.868 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:17.868 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:17.868 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:17.869 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:17.870 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:17.871 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:17.872 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:17.873 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:17.874 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:22.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:22.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:22.875 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:22.931 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:22.931 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:23.076 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:23.077 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:23.078 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:23.080 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:23.081 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:23.082 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:28.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:28.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:28.084 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:28.196 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:28.196 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:28.196 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:28.197 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:28.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:28.199 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:28.200 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:28.201 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:28.203 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:33.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:33.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:33.204 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:33.255 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:33.311 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:33.311 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:33.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:33.314 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:33.318 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:33.320 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:33.321 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:38.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:38.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:38.323 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:38.374 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:38.374 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:38.437 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:38.437 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:38.437 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:38.437 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:38.437 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:38.437 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:38.438 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:38.439 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:38.440 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:38.442 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:38.443 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:43.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:43.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:43.445 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:43.495 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:43.495 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:43.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:43.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:43.566 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:43.567 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:43.568 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:43.569 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:43.569 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:43.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:48.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:48.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:48.571 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:48.623 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:48.624 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:48.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:48.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:48.756 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:48.757 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:48.758 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:48.759 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:48.759 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:48.759 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:48.760 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:53.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:53.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:53.761 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:53.806 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:53.806 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:53.875 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:53.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:53.877 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:53.878 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:53.879 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:53.880 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:53.880 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:53.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:15:58.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:58.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:15:58.882 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:15:58.933 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:15:58.933 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:15:58.998 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:15:58.998 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:15:58.998 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:15:58.998 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:15:58.998 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:15:58.999 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:59.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:15:59.001 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:59.002 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:15:59.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:15:59.003 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:15:59.003 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:15:59.003 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:15:59.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:15:59.003 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:15:59.004 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:04.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:04.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:04.005 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:04.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:04.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:04.132 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:04.133 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:04.134 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:04.135 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:04.135 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:04.136 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:09.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:09.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:09.137 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:09.188 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:09.188 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:09.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:09.271 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:09.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:09.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:09.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:09.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:09.275 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:09.276 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:14.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:14.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:14.278 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:14.407 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:14.407 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:14.407 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:14.408 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:14.409 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:14.410 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724281871.246527_404.1", 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:11:11.246527Z", 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:14.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:14.412 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:14.413 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:19.415 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2024-08-21T23:16:19.416 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-48 2024-08-21T23:16:19.443 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-49 2024-08-21T23:16:19.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:19.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:19.630 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:19.677 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:19.733 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T23:16:19.746 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T23:16:19.747 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T23:16:19.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.748 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.748 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:19.748 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T23:16:19.471809Z" 2024-08-21T23:16:19.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:19.748 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T23:16:19.749 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T23:16:19.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:19.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:19.749 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:19.794 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:19.794 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:19.854 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:19.855 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:19.856 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:19.857 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:19.858 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:19.859 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:19.860 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:24.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:24.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:24.861 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:24.906 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:24.980 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:24.980 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:24.980 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:24.980 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:24.980 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:24.980 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:24.981 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:24.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:24.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:24.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:24.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:24.985 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:24.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:29.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:29.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:29.987 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:30.034 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:30.035 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:30.085 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:30.085 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:30.086 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:30.086 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:30.086 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:30.086 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:30.086 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:30.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:30.087 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:30.087 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:30.087 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:30.087 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:30.087 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:30.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:30.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:30.088 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:30.088 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:30.088 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:30.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:30.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:30.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:30.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:30.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:30.090 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:30.090 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:30.090 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:30.091 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:30.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:30.091 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:30.091 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:30.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:30.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:30.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:30.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:30.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:30.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:30.092 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:30.092 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:30.093 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:30.093 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:30.093 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:30.093 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:30.093 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:30.094 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:30.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:30.094 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:30.094 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:30.094 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:30.095 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:30.095 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:30.095 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:30.095 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:30.095 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:30.095 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:30.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:30.097 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:30.097 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:30.099 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:35.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:35.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:35.101 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:35.153 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:35.154 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:35.238 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:35.239 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:40.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:40.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:40.241 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:40.362 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:40.363 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:40.364 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:40.365 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:40.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:40.367 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:40.368 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:40.369 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:45.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:45.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:45.370 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:45.423 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:45.423 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:45.484 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:45.485 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:45.486 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:45.487 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:45.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:45.489 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:45.489 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:45.489 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:45.489 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:45.489 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:45.490 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:50.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:50.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:50.491 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:50.536 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:50.621 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:50.622 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:50.622 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:50.622 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:50.622 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:50.622 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:50.622 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:50.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:50.623 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:50.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:50.623 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:50.623 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:50.623 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:50.623 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:50.624 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:50.624 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:50.624 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:50.624 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:50.624 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:50.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:50.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:50.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:50.625 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:50.625 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:50.625 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:50.626 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:50.627 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:50.628 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:50.629 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:50.630 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:16:55.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:55.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:16:55.634 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:16:55.680 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:16:55.680 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:16:55.760 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:55.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:16:55.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:16:55.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:16:55.764 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:16:55.765 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:16:55.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:00.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:00.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:00.768 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:00.954 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:00.954 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:00.954 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:00.954 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:00.954 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:00.954 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:00.955 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:00.956 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:00.957 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:00.958 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:00.961 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:05.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:05.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:05.963 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:06.011 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:06.067 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:06.067 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:06.068 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:06.069 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:06.070 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:06.071 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:06.072 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:06.073 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:11.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:11.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:11.075 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:11.187 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:11.187 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:11.188 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:11.189 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:11.190 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:11.191 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:11.192 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:11.193 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:16.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:16.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:16.195 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:16.246 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:16.246 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:16.317 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:16.317 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:16.317 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:16.317 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:16.317 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:16.317 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:16.318 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:16.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:16.321 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:16.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:16.322 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:16.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:16.322 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:16.323 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:21.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:21.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:21.325 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:21.444 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:21.444 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:21.444 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:21.444 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:21.444 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:21.444 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:21.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:21.446 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:21.448 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:21.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:21.449 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:21.451 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:26.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:26.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:26.453 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:26.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:26.601 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:26.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:26.603 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:26.603 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:26.603 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:26.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:26.603 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:26.604 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:31.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:31.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:31.605 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:31.652 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:31.652 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:31.717 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:31.717 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:31.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:31.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:31.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:31.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:31.722 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:31.722 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:31.723 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:36.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:36.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:36.725 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:36.774 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:36.774 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:36.842 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:36.842 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:36.842 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:36.842 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:36.843 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:36.844 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:36.845 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:36.846 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:36.847 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:36.847 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:36.847 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:36.847 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:36.847 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:36.847 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:36.848 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:36.849 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:41.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:41.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:41.851 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:41.961 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:41.961 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:41.962 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:41.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:41.964 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:41.965 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:41.966 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:41.966 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:41.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:41.966 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:41.967 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:46.968 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:46.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:46.968 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:47.020 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:47.020 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:47.087 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:47.087 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:47.088 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:47.089 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:47.090 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:47.091 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:47.092 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:47.093 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:52.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:52.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:52.095 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:52.145 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:52.145 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:52.202 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:52.203 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:52.204 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:52.205 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:52.207 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:52.208 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:52.209 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:17:57.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:57.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:17:57.211 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:17:57.262 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:17:57.262 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:17:57.337 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:17:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:17:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:17:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:17:57.341 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:17:57.342 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:17:57.343 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:02.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:02.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:02.345 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:02.394 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:02.394 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:02.475 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:02.566 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:02.567 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:07.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:07.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:07.568 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:07.620 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:07.621 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:07.693 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:07.694 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:07.695 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:07.696 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:07.697 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:07.698 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:07.699 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:12.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:12.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:12.701 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:12.752 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:12.753 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:12.836 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:12.836 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:12.838 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:12.839 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:12.840 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:12.841 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:12.842 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:17.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:17.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:17.843 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:17.895 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:17.956 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:17.956 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:17.956 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:17.956 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:17.957 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:17.957 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:17.957 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:17.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:17.957 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:17.957 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:17.957 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:17.958 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:17.959 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:17.960 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:17.961 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:17.962 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:17.963 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:17.963 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:17.963 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:17.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:22.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:22.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:22.966 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:23.017 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:23.099 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:23.099 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:23.099 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:23.099 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:23.100 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:23.101 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:23.102 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:23.103 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:23.105 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:23.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:23.105 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:23.106 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:28.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:28.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:28.107 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:28.158 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:28.159 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:28.284 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:28.284 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:28.284 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:28.284 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:28.284 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:28.284 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:28.284 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:28.285 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:28.286 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:28.287 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:28.288 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:28.289 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:28.290 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:33.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:33.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:33.353 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:33.405 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:33.405 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:33.492 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:33.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:33.494 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:33.495 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:33.496 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:33.497 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:33.497 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:33.497 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:33.497 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:33.498 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:38.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:38.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:38.500 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:38.620 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:38.620 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:38.620 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:38.620 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:38.620 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:38.620 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:38.621 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:38.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:38.621 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:38.621 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:38.621 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:38.621 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:38.621 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:38.622 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:38.622 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:38.622 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:38.622 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:38.622 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:38.622 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:38.623 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:38.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:38.623 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:38.623 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:38.623 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:38.623 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:38.623 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:38.624 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:38.624 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:38.624 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:38.624 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:38.624 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:38.624 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:38.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:38.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:38.626 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:38.627 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:38.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:38.627 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:38.627 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:38.627 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:38.627 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:38.627 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:38.628 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:43.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:43.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:43.630 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:43.682 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:43.760 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:43.761 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:43.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:43.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:43.764 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:43.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:43.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:43.765 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:43.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:43.765 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:43.765 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:43.766 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:43.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:43.766 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:43.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:48.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:48.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:48.768 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:48.900 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:48.900 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:48.900 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:48.900 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:48.901 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:48.902 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:48.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:48.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:48.905 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:48.905 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:48.905 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:48.905 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:48.905 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:48.906 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:53.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:53.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:53.908 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:53.958 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:54.014 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:54.014 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:54.015 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:54.016 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:54.017 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:54.018 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:54.019 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:54.020 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:54.021 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:54.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:54.023 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:54.024 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:54.025 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:54.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:54.026 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:54.028 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:18:59.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:59.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:18:59.029 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:18:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:18:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:59.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:59.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:59.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:59.152 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:18:59.153 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:18:59.154 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:18:59.154 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:18:59.154 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:18:59.154 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:18:59.155 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:04.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:04.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:04.157 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:04.207 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:04.207 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:04.306 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:04.306 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:04.306 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:04.306 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:04.306 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:04.307 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:04.308 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:04.309 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:04.310 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:04.311 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:04.312 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:04.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:09.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:09.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:09.314 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:09.365 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:09.366 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:09.449 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:09.451 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:09.452 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:09.453 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:09.454 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:09.455 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:14.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:14.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:14.456 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:14.508 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:14.508 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:14.584 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:14.584 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:14.585 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:14.586 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:14.587 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:14.588 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:14.589 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:14.589 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:14.590 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:19.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:19.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:19.591 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:19.714 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:19.714 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:19.715 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:19.716 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:19.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:19.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:19.719 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:19.720 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:24.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:24.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:24.722 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:24.773 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:24.773 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:24.834 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:24.834 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:24.835 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:24.836 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:24.837 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:24.838 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:24.839 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:24.840 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:29.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:29.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:29.842 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:29.893 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:29.894 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:30.026 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:30.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:30.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:30.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:30.030 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:30.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:30.031 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:30.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:35.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:35.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:35.034 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:35.078 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:35.153 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:35.159 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:35.160 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:40.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:40.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:40.161 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:40.295 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:40.296 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:40.297 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:40.298 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:40.299 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:40.300 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:40.301 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:45.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:45.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:45.302 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:45.353 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:45.353 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:45.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:45.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:45.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:45.438 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:45.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:45.440 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:45.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:45.440 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:45.440 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:45.440 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:45.440 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:45.440 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:45.441 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:50.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:50.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:50.443 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:50.573 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:50.573 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:50.573 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:50.573 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:50.573 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:50.574 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:50.575 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:50.576 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:50.577 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:50.578 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:50.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:19:55.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:55.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:19:55.581 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:19:55.648 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:19:55.648 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:19:55.784 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:19:55.784 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:19:55.784 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:55.785 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:55.786 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:55.787 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:19:55.788 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:19:55.789 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:19:55.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:00.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:00.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:00.792 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:00.889 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:00.889 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:00.889 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:00.889 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:00.889 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:00.890 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:00.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:00.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:00.893 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:00.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:00.894 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:00.894 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:00.894 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:00.894 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:00.894 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:00.895 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:05.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:05.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:05.897 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:06.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:06.014 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:06.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:06.016 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:06.017 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:06.018 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:06.018 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:06.018 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:06.019 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:11.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:11.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:11.021 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:11.068 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:11.135 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:11.135 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:11.135 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:11.135 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:11.135 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:11.135 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:11.136 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:11.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:11.138 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:11.139 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:11.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:11.140 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:11.141 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:16.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:16.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:16.142 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:16.192 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:16.192 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:16.247 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:16.248 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:16.250 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:16.251 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:16.252 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:16.253 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:21.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:21.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:21.254 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:21.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:21.372 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:21.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:21.374 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:21.375 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:21.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:21.376 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:21.377 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:26.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:26.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:26.378 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:26.428 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:26.429 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:26.490 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:26.490 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:26.490 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:26.490 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:26.490 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:26.491 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:26.492 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:26.493 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:26.494 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:26.495 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:26.496 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:31.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:31.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:31.498 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:31.549 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:31.605 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:31.605 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:31.605 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:31.605 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:31.606 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:31.607 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:31.608 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:31.609 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:31.610 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:31.611 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:36.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:36.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:36.613 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:36.663 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:36.737 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:36.737 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:36.737 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:36.737 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:36.738 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:36.739 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:36.740 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:36.741 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:36.742 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:36.742 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:36.743 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:41.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:41.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:41.745 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:41.796 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:41.796 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:41.861 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:41.863 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:41.864 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:41.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:41.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:41.867 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:41.867 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:46.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:46.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:46.869 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:46.920 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:46.920 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:47.007 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:47.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:47.010 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:47.011 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:47.012 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:47.013 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:47.014 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:52.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:52.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:52.015 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:52.067 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:52.067 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:52.145 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:52.145 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:52.145 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:52.146 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:52.147 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:52.148 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:52.149 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:52.150 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:52.152 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:20:57.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:57.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:20:57.154 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:20:57.202 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:20:57.203 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:20:57.335 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:20:57.335 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:20:57.335 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:20:57.335 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:20:57.336 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:57.338 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:57.339 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:20:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:20:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:20:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:20:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:20:57.340 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:20:57.340 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:20:57.341 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:02.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:02.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:02.342 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:02.387 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:02.446 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:02.447 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:07.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:07.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:07.449 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:07.500 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:07.500 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:07.581 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:07.581 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:07.582 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:07.583 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:07.584 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:07.585 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:07.586 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:12.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:12.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:12.587 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:12.638 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:12.704 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:12.704 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:12.704 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:12.704 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:12.704 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:12.704 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:12.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:12.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:12.707 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:12.708 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:12.709 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:17.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:17.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:17.710 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:17.761 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:17.830 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:17.830 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:17.830 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:17.831 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:17.832 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:17.833 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:21:17.834 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:17.835 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:17.836 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:17.836 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:17.837 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:22.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:22.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:22.838 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:22.889 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:22.889 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282179.471809_426.1", 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:16:19.471809Z", 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:22.973 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:22.973 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:27.975 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2024-08-21T23:21:27.976 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-50 2024-08-21T23:21:28.007 INFO:rgw_multi.tests:create bucket zone=a2 name=qlihov-51 2024-08-21T23:21:30.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qlihov-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-21T23:21:30.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qlihov-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-21T23:21:30.394 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qlihov-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-21T23:21:30.445 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:30.445 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:30.508 INFO:teuthology.orchestra.run.smithi116.stdout:Sync is disabled for bucket qlihov-50 2024-08-21T23:21:30.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'qlihov-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-21T23:21:30.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'qlihov-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2024-08-21T23:21:30.527 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket qlihov-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2024-08-21T23:21:30.577 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:30.577 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:30.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T23:21:30.620+0000 7fb6dbd8b7c0 1 bucket sync caught up with empty source 2024-08-21T23:21:30.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-21T23:21:30.620+0000 7fb6dbd8b7c0 0 bucket checkpoint complete 2024-08-21T23:21:30.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=qlihov-50 zones={a2, a1} 2024-08-21T23:21:30.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2024-08-21T23:21:30.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2024-08-21T23:21:30.644 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2024-08-21T23:21:30.649 CRITICAL:tasks.rgw_multi.zone_rados:key=MULTIPART is missing from zone=a2 2024-08-21T23:21:30.650 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... FAIL 2024-08-21T23:21:30.650 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-52 2024-08-21T23:21:30.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:30.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:30.732 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:30.774 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:No --period given, using current period=2584ad6b-0698-496d-8897-87f74ca5f014 2024-08-21T23:21:30.865 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-08-21T23:21:30.865 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:30.865 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:30.865 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:01:02.145957Z" 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "0.000000" 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724278411.514728_132.1", 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T22:13:31.514728Z" 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: "last_update": "2024-08-21T23:21:30.663596Z" 2024-08-21T23:21:30.866 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:30.867 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-08-21T23:21:30.867 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2024-08-21T23:21:30.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:30.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:30.868 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:30.951 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:31.015 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:31.016 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:31.017 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:31.018 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:31.019 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:31.020 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:31.023 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:36.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:36.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:36.025 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:36.076 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:36.076 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:36.153 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:36.153 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:36.154 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:36.155 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:36.156 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:21:36.157 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:36.158 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:36.159 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:41.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:41.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:41.160 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:41.210 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:41.210 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:41.292 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:41.293 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:41.294 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:41.295 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:41.296 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:41.297 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:46.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:46.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:46.299 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:46.346 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:46.346 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:46.411 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:46.411 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:46.411 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:46.411 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:46.411 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:46.411 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:46.412 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:46.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:46.415 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:46.416 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:51.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:51.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:51.418 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:51.464 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:51.464 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:51.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:51.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:51.542 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:51.543 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:51.544 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:51.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:21:56.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:56.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:21:56.547 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:21:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:21:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:21:56.666 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:21:56.667 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:56.668 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:21:56.669 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:21:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:21:56.671 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:21:56.672 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:01.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:01.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:01.674 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:01.788 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:01.789 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:06.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:06.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:06.790 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:06.842 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:06.842 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:06.920 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:06.921 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:06.922 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:06.923 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:06.924 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:06.925 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:06.926 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:11.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:11.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:11.928 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:11.978 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:12.054 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:12.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:12.056 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:12.057 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:12.058 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:12.059 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:12.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:12.059 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:12.060 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:17.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:17.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:17.060 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:17.105 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:17.174 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:17.175 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:17.176 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:17.177 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:17.178 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:17.179 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:17.180 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:22.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:22.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:22.181 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:22.232 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:22.310 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:22.311 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:22.312 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:22.313 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:22.314 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:22.315 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:22.316 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:22.317 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:27.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:27.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:27.318 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:27.438 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:27.438 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:27.438 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:27.438 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:27.438 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:27.438 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:27.438 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:27.439 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:27.440 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:27.441 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:27.442 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:27.443 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:27.444 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:27.444 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:32.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:32.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:32.446 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:32.559 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:32.559 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:32.559 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:32.559 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:32.560 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:32.561 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:32.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:32.563 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:32.564 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:32.564 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:32.564 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:32.565 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:37.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:37.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:37.567 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:37.618 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:37.618 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:37.701 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:37.701 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:37.701 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:37.701 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:37.701 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:37.701 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:37.701 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:37.702 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:37.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:37.704 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:37.705 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:37.706 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:37.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:37.706 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:37.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:42.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:42.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:42.708 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:42.753 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:42.753 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:42.819 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:42.820 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:42.821 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:42.822 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:42.823 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:42.824 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:47.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:47.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:47.826 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:47.894 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:47.895 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:48.026 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:48.026 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:48.026 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:48.027 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:48.028 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:48.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:48.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:48.031 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:48.031 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:48.031 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:48.031 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:48.031 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:48.032 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:53.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:53.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:53.034 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:53.079 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:53.079 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:53.150 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:53.151 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:53.152 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:53.153 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:53.154 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:53.155 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:53.156 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:22:58.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:58.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:22:58.157 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:22:58.207 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:22:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:22:58.268 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:22:58.268 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:58.269 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:58.270 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:22:58.271 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:58.272 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:22:58.273 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:22:58.274 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:22:58.275 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:22:58.276 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:03.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:03.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:03.277 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:03.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:03.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:03.407 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:03.409 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:03.410 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:03.410 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:03.411 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:08.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:08.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:08.413 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:08.531 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:08.531 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:08.531 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:08.531 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:08.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:08.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:08.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:08.535 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:08.536 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:08.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:08.536 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:08.537 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:13.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:13.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:13.538 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:13.600 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:13.600 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:13.729 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:13.729 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:13.729 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:13.729 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:13.729 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:13.729 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:13.730 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:13.731 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:13.732 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:13.733 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:13.734 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:13.736 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:18.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:18.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:18.737 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:18.786 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:18.787 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:18.848 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:18.848 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:18.848 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:18.848 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:18.848 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:18.848 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:18.848 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:18.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:18.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:18.851 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:18.852 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:18.853 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:18.853 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:18.854 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:23.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:23.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:23.855 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:23.906 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:23.906 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:23.965 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:23.965 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:23.966 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:23.967 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:23.968 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:23.969 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:23.970 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:23.971 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:28.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:28.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:28.972 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:29.022 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:29.090 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:29.091 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:29.092 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:29.093 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:29.094 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:29.095 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:29.095 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:34.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:34.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:34.098 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:34.149 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:34.149 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:34.211 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:34.212 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:39.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:39.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:39.213 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:39.283 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:39.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:39.394 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:39.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:39.396 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:39.397 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:39.398 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:39.399 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:44.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:44.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:44.400 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:44.510 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:44.511 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:44.512 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:44.513 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:44.514 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:44.515 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:49.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:49.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:49.517 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:49.566 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:49.566 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:49.636 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:49.637 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:49.638 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:49.639 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:49.640 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:49.641 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:49.641 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:49.642 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:54.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:54.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:54.643 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:54.761 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:54.762 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:54.763 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:54.764 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:54.766 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:54.767 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:23:59.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:59.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:23:59.769 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:23:59.819 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:23:59.819 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:23:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:23:59.889 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:59.891 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:23:59.892 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:23:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:23:59.894 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:04.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:04.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:04.896 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:05.021 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:05.022 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:05.023 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:05.024 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:05.025 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:05.026 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:05.027 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:10.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:10.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:10.029 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:10.146 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:10.147 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:10.148 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:10.149 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:10.150 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:10.151 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:10.152 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:15.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:15.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:15.154 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:15.205 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:15.205 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:15.278 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:15.278 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:15.279 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:15.280 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:15.281 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:15.282 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:15.283 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:20.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:20.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:20.285 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:20.336 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:20.336 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:20.426 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:20.427 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:20.429 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:20.430 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:20.431 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:25.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:25.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:25.432 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:25.477 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:25.477 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:25.530 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:25.530 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:25.530 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:25.530 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:25.530 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:25.530 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:25.531 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:25.532 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:25.533 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:25.534 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:25.535 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:25.535 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:25.536 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:30.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:30.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:30.537 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:30.609 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:30.715 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:30.715 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:30.716 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:30.717 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:30.718 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:30.719 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:30.720 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:30.721 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:35.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:35.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:35.723 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:35.774 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:35.775 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:35.845 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:35.850 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:35.851 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:40.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:40.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:40.853 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:40.901 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:40.901 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:40.979 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:40.979 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:40.979 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:40.979 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:40.980 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:40.980 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:40.980 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:40.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:40.980 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:40.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:40.980 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:40.981 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:40.981 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:40.981 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:40.981 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:40.981 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:40.982 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:40.982 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:40.982 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:40.982 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:40.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:40.982 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:40.983 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:40.983 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:40.983 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:40.983 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:40.983 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:40.983 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:40.983 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:40.984 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:40.984 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:40.984 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:40.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:40.984 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:40.984 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:40.984 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:40.985 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:40.986 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:40.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:40.986 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:40.986 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:40.986 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:40.986 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:40.987 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:40.987 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:40.987 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:40.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:40.987 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:40.987 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:40.987 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:40.988 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:40.988 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:40.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:40.988 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:40.988 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:40.988 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:40.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:40.989 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:40.990 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:45.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:45.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:45.992 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:46.043 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:46.043 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:46.121 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:46.121 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:46.121 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:46.121 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:46.121 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:46.121 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:46.122 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:46.123 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:46.124 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:46.125 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:46.126 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:46.126 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:46.126 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:46.127 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:51.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:51.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:51.128 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:51.179 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:51.179 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:51.255 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:51.256 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:51.257 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:51.258 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:51.260 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:24:56.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:56.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:24:56.262 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:24:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:24:56.326 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:56.429 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:56.430 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:56.431 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:24:56.432 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:24:56.433 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:24:56.434 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:01.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:01.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:01.435 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:25:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:25:01.539 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:25:01.539 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:25:01.539 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:25:01.539 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:25:01.539 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:25:01.539 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:01.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:01.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:01.542 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:01.543 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:25:01.545 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:06.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:06.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:06.547 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:06.598 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:25:06.598 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:25:06.669 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:06.670 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:25:06.671 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:06.672 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:25:06.673 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:25:06.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:06.674 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:25:06.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:11.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:11.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:11.676 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:11.725 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:25:11.726 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:25:11.793 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:25:11.793 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:25:11.793 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:25:11.793 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:25:11.793 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:25:11.793 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:25:11.793 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:11.794 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:11.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:11.796 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:11.797 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:25:11.798 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:25:11.798 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:25:11.798 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:11.798 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:25:11.799 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:16.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:16.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:16.801 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:16.851 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:25:16.851 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:16.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:16.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:16.925 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:16.926 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:25:16.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:16.928 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:25:16.928 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:25:16.928 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:25:16.928 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:16.928 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:25:16.929 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:21.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:21.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:21.931 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:21.982 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:25:21.982 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:25:22.057 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:25:22.057 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:25:22.057 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:22.058 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:22.059 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:22.060 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:22.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:22.061 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:22.061 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:25:22.061 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:22.061 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:22.061 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:25:22.061 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:22.062 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:22.063 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:25:22.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:22.064 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:25:22.064 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:25:22.064 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:25:22.064 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:22.064 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:25:22.065 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:27.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:27.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:27.067 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:25:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:25:27.189 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:27.190 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:27.191 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:25:27.192 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:25:27.193 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:25:27.194 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:25:27.194 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:27.194 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:25:27.195 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:32.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:32.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:32.196 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:25:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:25:32.318 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-21T23:25:32.318 INFO:teuthology.orchestra.run.smithi116.stdout: "sync_status": { 2024-08-21T23:25:32.318 INFO:teuthology.orchestra.run.smithi116.stdout: "info": { 2024-08-21T23:25:32.318 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "sync", 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "num_shards": 4, 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "period": "2584ad6b-0698-496d-8897-87f74ca5f014", 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "markers": [ 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 0, 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277662.145957_63.1", 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 1, 2024-08-21T23:25:32.319 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:01:02.145957Z", 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 1, 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "", 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "0.000000", 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:32.320 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 2, 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724277797.726665_94.1", 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T22:03:17.726665Z", 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-08-21T23:25:32.321 INFO:teuthology.orchestra.run.smithi116.stdout: "key": 3, 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "val": { 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "state": 1, 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "marker": "1_1724282490.663596_448.1", 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "next_step_marker": "", 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "total_entries": 0, 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "pos": 0, 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "timestamp": "2024-08-21T23:21:30.663596Z", 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "realm_epoch": 2 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-08-21T23:25:32.322 INFO:teuthology.orchestra.run.smithi116.stdout: "full_sync": { 2024-08-21T23:25:32.323 INFO:teuthology.orchestra.run.smithi116.stdout: "total": 1, 2024-08-21T23:25:32.323 INFO:teuthology.orchestra.run.smithi116.stdout: "complete": 1 2024-08-21T23:25:32.323 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-08-21T23:25:32.323 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-21T23:25:32.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=2 master=1_1724278411.514728_132.1 target=1_1724277797.726665_94.1 2024-08-21T23:25:37.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:37.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:25:37.326 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:25:37.327 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-08-21T23:25:37.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2024-08-21T23:25:37.331 INFO:rgw_multi.tests:create bucket zone=a1 name=qlihov-53 2024-08-21T23:25:41.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~0s 2024-08-21T23:25:41.443 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~0s 2024-08-21T23:25:41.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~0s 2024-08-21T23:25:41.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~0s 2024-08-21T23:25:41.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2024-08-21T23:25:41.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2024-08-21T23:25:41.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~0s 2024-08-21T23:25:46.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~5s 2024-08-21T23:25:46.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~5s 2024-08-21T23:25:46.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~5s 2024-08-21T23:25:46.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~5s 2024-08-21T23:25:46.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~5s 2024-08-21T23:25:46.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~5s 2024-08-21T23:25:46.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~5s 2024-08-21T23:25:51.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~10s 2024-08-21T23:25:51.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~10s 2024-08-21T23:25:51.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~10s 2024-08-21T23:25:51.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~10s 2024-08-21T23:25:51.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~10s 2024-08-21T23:25:51.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~10s 2024-08-21T23:25:51.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~10s 2024-08-21T23:25:56.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~15s 2024-08-21T23:25:56.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~15s 2024-08-21T23:25:56.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~15s 2024-08-21T23:25:56.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~15s 2024-08-21T23:25:56.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~15s 2024-08-21T23:25:56.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~15s 2024-08-21T23:25:56.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~15s 2024-08-21T23:26:01.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~20s 2024-08-21T23:26:01.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~20s 2024-08-21T23:26:01.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~20s 2024-08-21T23:26:01.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~20s 2024-08-21T23:26:01.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~20s 2024-08-21T23:26:01.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~20s 2024-08-21T23:26:01.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~20s 2024-08-21T23:26:06.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~25s 2024-08-21T23:26:06.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~25s 2024-08-21T23:26:06.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~25s 2024-08-21T23:26:06.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~25s 2024-08-21T23:26:06.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~25s 2024-08-21T23:26:06.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~25s 2024-08-21T23:26:06.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~25s 2024-08-21T23:26:11.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~30s 2024-08-21T23:26:11.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~30s 2024-08-21T23:26:11.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~30s 2024-08-21T23:26:11.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~30s 2024-08-21T23:26:11.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~30s 2024-08-21T23:26:11.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~30s 2024-08-21T23:26:11.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~30s 2024-08-21T23:26:16.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~35s 2024-08-21T23:26:16.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~35s 2024-08-21T23:26:16.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~35s 2024-08-21T23:26:16.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~35s 2024-08-21T23:26:16.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~35s 2024-08-21T23:26:16.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~35s 2024-08-21T23:26:16.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~35s 2024-08-21T23:26:21.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~40s 2024-08-21T23:26:21.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~40s 2024-08-21T23:26:21.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~40s 2024-08-21T23:26:21.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~40s 2024-08-21T23:26:21.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~40s 2024-08-21T23:26:21.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~40s 2024-08-21T23:26:21.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~40s 2024-08-21T23:26:26.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~45s 2024-08-21T23:26:26.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~45s 2024-08-21T23:26:26.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~45s 2024-08-21T23:26:26.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~45s 2024-08-21T23:26:26.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2024-08-21T23:26:26.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~45s 2024-08-21T23:26:26.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~45s 2024-08-21T23:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~50s 2024-08-21T23:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~50s 2024-08-21T23:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~50s 2024-08-21T23:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~50s 2024-08-21T23:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2024-08-21T23:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~50s 2024-08-21T23:26:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~50s 2024-08-21T23:26:36.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~55s 2024-08-21T23:26:36.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~55s 2024-08-21T23:26:36.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~55s 2024-08-21T23:26:36.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~55s 2024-08-21T23:26:36.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~55s 2024-08-21T23:26:36.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~55s 2024-08-21T23:26:36.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~55s 2024-08-21T23:26:41.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~60s 2024-08-21T23:26:41.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~60s 2024-08-21T23:26:41.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~60s 2024-08-21T23:26:41.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~60s 2024-08-21T23:26:41.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~60s 2024-08-21T23:26:41.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~60s 2024-08-21T23:26:41.464 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~60s 2024-08-21T23:26:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~65s 2024-08-21T23:26:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~65s 2024-08-21T23:26:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~65s 2024-08-21T23:26:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~65s 2024-08-21T23:26:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~65s 2024-08-21T23:26:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~65s 2024-08-21T23:26:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~65s 2024-08-21T23:26:49.133 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2024-08-21T23:26:49.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:26:49.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2024-08-21T23:26:49.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-21T23:26:49.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-21T23:26:49.340 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-08-21T23:26:49.705 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-21T23:26:49.705 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2024-08-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setuser ceph since I am not root 2024-08-21T23:26:49.826 INFO:teuthology.orchestra.run.smithi116.stderr:ignoring --setgroup ceph since I am not root 2024-08-21T23:26:51.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~70s 2024-08-21T23:26:51.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~70s 2024-08-21T23:26:51.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~70s 2024-08-21T23:26:51.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~70s 2024-08-21T23:26:51.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~70s 2024-08-21T23:26:51.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~70s 2024-08-21T23:26:51.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~70s 2024-08-21T23:26:56.467 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~75s 2024-08-21T23:26:56.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~75s 2024-08-21T23:26:56.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~75s 2024-08-21T23:26:56.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~75s 2024-08-21T23:26:56.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~75s 2024-08-21T23:26:56.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~75s 2024-08-21T23:26:56.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~75s 2024-08-21T23:27:01.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~80s 2024-08-21T23:27:01.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~80s 2024-08-21T23:27:01.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~80s 2024-08-21T23:27:01.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~80s 2024-08-21T23:27:01.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~80s 2024-08-21T23:27:01.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~80s 2024-08-21T23:27:01.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~80s 2024-08-21T23:27:06.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~85s 2024-08-21T23:27:06.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~85s 2024-08-21T23:27:06.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~85s 2024-08-21T23:27:06.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~85s 2024-08-21T23:27:06.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~85s 2024-08-21T23:27:06.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~85s 2024-08-21T23:27:06.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~85s 2024-08-21T23:27:11.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~90s 2024-08-21T23:27:11.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~90s 2024-08-21T23:27:11.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~90s 2024-08-21T23:27:11.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~90s 2024-08-21T23:27:11.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2024-08-21T23:27:11.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~90s 2024-08-21T23:27:11.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~90s 2024-08-21T23:27:16.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~95s 2024-08-21T23:27:16.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~95s 2024-08-21T23:27:16.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~95s 2024-08-21T23:27:16.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~95s 2024-08-21T23:27:16.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2024-08-21T23:27:16.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~95s 2024-08-21T23:27:16.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~95s 2024-08-21T23:27:21.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~100s 2024-08-21T23:27:21.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~100s 2024-08-21T23:27:21.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~100s 2024-08-21T23:27:21.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~100s 2024-08-21T23:27:21.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~100s 2024-08-21T23:27:21.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~100s 2024-08-21T23:27:21.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~100s 2024-08-21T23:27:26.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~105s 2024-08-21T23:27:26.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~105s 2024-08-21T23:27:26.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~105s 2024-08-21T23:27:26.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~105s 2024-08-21T23:27:26.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~105s 2024-08-21T23:27:26.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~105s 2024-08-21T23:27:26.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~105s 2024-08-21T23:27:31.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~110s 2024-08-21T23:27:31.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~110s 2024-08-21T23:27:31.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~110s 2024-08-21T23:27:31.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~110s 2024-08-21T23:27:31.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~110s 2024-08-21T23:27:31.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~110s 2024-08-21T23:27:31.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~110s 2024-08-21T23:27:36.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~115s 2024-08-21T23:27:36.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~115s 2024-08-21T23:27:36.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~115s 2024-08-21T23:27:36.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~115s 2024-08-21T23:27:36.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~115s 2024-08-21T23:27:36.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~115s 2024-08-21T23:27:36.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~115s 2024-08-21T23:27:41.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~120s 2024-08-21T23:27:41.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~120s 2024-08-21T23:27:41.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~120s 2024-08-21T23:27:41.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~120s 2024-08-21T23:27:41.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~120s 2024-08-21T23:27:41.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~120s 2024-08-21T23:27:41.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~120s 2024-08-21T23:27:46.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~125s 2024-08-21T23:27:46.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~125s 2024-08-21T23:27:46.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~125s 2024-08-21T23:27:46.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~125s 2024-08-21T23:27:46.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~125s 2024-08-21T23:27:46.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~125s 2024-08-21T23:27:46.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~125s 2024-08-21T23:27:51.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~130s 2024-08-21T23:27:51.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~130s 2024-08-21T23:27:51.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~130s 2024-08-21T23:27:51.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~130s 2024-08-21T23:27:51.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~130s 2024-08-21T23:27:51.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~130s 2024-08-21T23:27:51.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~130s 2024-08-21T23:27:56.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~135s 2024-08-21T23:27:56.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~135s 2024-08-21T23:27:56.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~135s 2024-08-21T23:27:56.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~135s 2024-08-21T23:27:56.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2024-08-21T23:27:56.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~135s 2024-08-21T23:27:56.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~135s 2024-08-21T23:28:01.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~140s 2024-08-21T23:28:01.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~140s 2024-08-21T23:28:01.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~140s 2024-08-21T23:28:01.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~140s 2024-08-21T23:28:01.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2024-08-21T23:28:01.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~140s 2024-08-21T23:28:01.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~140s 2024-08-21T23:28:06.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~145s 2024-08-21T23:28:06.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~145s 2024-08-21T23:28:06.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~145s 2024-08-21T23:28:06.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~145s 2024-08-21T23:28:06.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~145s 2024-08-21T23:28:06.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~145s 2024-08-21T23:28:06.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~145s 2024-08-21T23:28:11.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~150s 2024-08-21T23:28:11.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~150s 2024-08-21T23:28:11.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~150s 2024-08-21T23:28:11.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~150s 2024-08-21T23:28:11.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~150s 2024-08-21T23:28:11.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~150s 2024-08-21T23:28:11.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~150s 2024-08-21T23:28:16.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~155s 2024-08-21T23:28:16.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~155s 2024-08-21T23:28:16.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~155s 2024-08-21T23:28:16.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~155s 2024-08-21T23:28:16.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~155s 2024-08-21T23:28:16.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~155s 2024-08-21T23:28:16.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~155s 2024-08-21T23:28:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~160s 2024-08-21T23:28:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~160s 2024-08-21T23:28:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~160s 2024-08-21T23:28:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~160s 2024-08-21T23:28:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~160s 2024-08-21T23:28:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~160s 2024-08-21T23:28:21.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~160s 2024-08-21T23:28:26.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~165s 2024-08-21T23:28:26.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~165s 2024-08-21T23:28:26.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~165s 2024-08-21T23:28:26.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~165s 2024-08-21T23:28:26.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~165s 2024-08-21T23:28:26.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~165s 2024-08-21T23:28:26.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~165s 2024-08-21T23:28:31.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~170s 2024-08-21T23:28:31.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~170s 2024-08-21T23:28:31.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~170s 2024-08-21T23:28:31.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~170s 2024-08-21T23:28:31.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~170s 2024-08-21T23:28:31.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~170s 2024-08-21T23:28:31.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~170s 2024-08-21T23:28:36.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~175s 2024-08-21T23:28:36.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~175s 2024-08-21T23:28:36.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~175s 2024-08-21T23:28:36.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~175s 2024-08-21T23:28:36.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~175s 2024-08-21T23:28:36.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~175s 2024-08-21T23:28:36.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~175s 2024-08-21T23:28:41.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~180s 2024-08-21T23:28:41.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~180s 2024-08-21T23:28:41.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~180s 2024-08-21T23:28:41.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~180s 2024-08-21T23:28:41.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~180s 2024-08-21T23:28:41.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~180s 2024-08-21T23:28:41.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~180s 2024-08-21T23:28:46.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~185s 2024-08-21T23:28:46.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~185s 2024-08-21T23:28:46.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~185s 2024-08-21T23:28:46.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~185s 2024-08-21T23:28:46.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2024-08-21T23:28:46.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~185s 2024-08-21T23:28:46.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~185s 2024-08-21T23:28:51.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~190s 2024-08-21T23:28:51.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~190s 2024-08-21T23:28:51.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~190s 2024-08-21T23:28:51.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~190s 2024-08-21T23:28:51.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~190s 2024-08-21T23:28:51.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~190s 2024-08-21T23:28:51.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~190s 2024-08-21T23:28:56.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~195s 2024-08-21T23:28:56.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~195s 2024-08-21T23:28:56.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~195s 2024-08-21T23:28:56.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~195s 2024-08-21T23:28:56.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~195s 2024-08-21T23:28:56.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~195s 2024-08-21T23:28:56.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~195s 2024-08-21T23:29:01.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~200s 2024-08-21T23:29:01.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~200s 2024-08-21T23:29:01.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~200s 2024-08-21T23:29:01.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~200s 2024-08-21T23:29:01.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~200s 2024-08-21T23:29:01.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~200s 2024-08-21T23:29:01.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~200s 2024-08-21T23:29:06.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~205s 2024-08-21T23:29:06.520 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~205s 2024-08-21T23:29:06.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~205s 2024-08-21T23:29:06.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~205s 2024-08-21T23:29:06.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~205s 2024-08-21T23:29:06.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~205s 2024-08-21T23:29:06.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~205s 2024-08-21T23:29:11.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~210s 2024-08-21T23:29:11.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~210s 2024-08-21T23:29:11.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~210s 2024-08-21T23:29:11.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~210s 2024-08-21T23:29:11.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~210s 2024-08-21T23:29:11.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~210s 2024-08-21T23:29:11.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~210s 2024-08-21T23:29:16.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~215s 2024-08-21T23:29:16.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~215s 2024-08-21T23:29:16.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~215s 2024-08-21T23:29:16.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~215s 2024-08-21T23:29:16.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~215s 2024-08-21T23:29:16.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~215s 2024-08-21T23:29:16.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~215s 2024-08-21T23:29:21.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~220s 2024-08-21T23:29:21.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~220s 2024-08-21T23:29:21.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~220s 2024-08-21T23:29:21.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~220s 2024-08-21T23:29:21.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~220s 2024-08-21T23:29:21.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~220s 2024-08-21T23:29:21.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~220s 2024-08-21T23:29:26.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~225s 2024-08-21T23:29:26.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~225s 2024-08-21T23:29:26.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~225s 2024-08-21T23:29:26.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~225s 2024-08-21T23:29:26.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~225s 2024-08-21T23:29:26.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~225s 2024-08-21T23:29:26.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~225s 2024-08-21T23:29:31.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~230s 2024-08-21T23:29:31.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~230s 2024-08-21T23:29:31.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~230s 2024-08-21T23:29:31.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~230s 2024-08-21T23:29:31.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2024-08-21T23:29:31.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~230s 2024-08-21T23:29:31.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~230s 2024-08-21T23:29:36.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~235s 2024-08-21T23:29:36.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~235s 2024-08-21T23:29:36.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~235s 2024-08-21T23:29:36.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~235s 2024-08-21T23:29:36.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2024-08-21T23:29:36.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~235s 2024-08-21T23:29:36.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~235s 2024-08-21T23:29:41.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~240s 2024-08-21T23:29:41.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~240s 2024-08-21T23:29:41.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~240s 2024-08-21T23:29:41.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~240s 2024-08-21T23:29:41.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~240s 2024-08-21T23:29:41.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~240s 2024-08-21T23:29:41.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~240s 2024-08-21T23:29:46.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~245s 2024-08-21T23:29:46.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~245s 2024-08-21T23:29:46.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~245s 2024-08-21T23:29:46.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~245s 2024-08-21T23:29:46.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~245s 2024-08-21T23:29:46.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~245s 2024-08-21T23:29:46.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~245s 2024-08-21T23:29:51.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~250s 2024-08-21T23:29:51.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~250s 2024-08-21T23:29:51.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~250s 2024-08-21T23:29:51.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~250s 2024-08-21T23:29:51.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~250s 2024-08-21T23:29:51.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~250s 2024-08-21T23:29:51.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~250s 2024-08-21T23:29:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~255s 2024-08-21T23:29:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~255s 2024-08-21T23:29:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~255s 2024-08-21T23:29:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~255s 2024-08-21T23:29:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~255s 2024-08-21T23:29:56.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~255s 2024-08-21T23:29:56.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~255s 2024-08-21T23:30:01.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~260s 2024-08-21T23:30:01.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~260s 2024-08-21T23:30:01.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~260s 2024-08-21T23:30:01.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~260s 2024-08-21T23:30:01.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~260s 2024-08-21T23:30:01.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~260s 2024-08-21T23:30:01.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~260s 2024-08-21T23:30:06.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~265s 2024-08-21T23:30:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~265s 2024-08-21T23:30:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~265s 2024-08-21T23:30:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~265s 2024-08-21T23:30:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~265s 2024-08-21T23:30:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~265s 2024-08-21T23:30:06.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~265s 2024-08-21T23:30:11.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~270s 2024-08-21T23:30:11.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~270s 2024-08-21T23:30:11.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~270s 2024-08-21T23:30:11.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~270s 2024-08-21T23:30:11.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~270s 2024-08-21T23:30:11.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~270s 2024-08-21T23:30:11.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~270s 2024-08-21T23:30:16.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~275s 2024-08-21T23:30:16.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~275s 2024-08-21T23:30:16.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~275s 2024-08-21T23:30:16.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~275s 2024-08-21T23:30:16.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2024-08-21T23:30:16.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~275s 2024-08-21T23:30:16.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~275s 2024-08-21T23:30:21.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~280s 2024-08-21T23:30:21.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~280s 2024-08-21T23:30:21.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~280s 2024-08-21T23:30:21.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~280s 2024-08-21T23:30:21.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2024-08-21T23:30:21.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~280s 2024-08-21T23:30:21.547 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~280s 2024-08-21T23:30:26.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~285s 2024-08-21T23:30:26.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~285s 2024-08-21T23:30:26.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~285s 2024-08-21T23:30:26.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~285s 2024-08-21T23:30:26.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~285s 2024-08-21T23:30:26.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~285s 2024-08-21T23:30:26.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~285s 2024-08-21T23:30:31.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~290s 2024-08-21T23:30:31.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~290s 2024-08-21T23:30:31.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~290s 2024-08-21T23:30:31.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~290s 2024-08-21T23:30:31.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~290s 2024-08-21T23:30:31.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~290s 2024-08-21T23:30:31.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~290s 2024-08-21T23:30:36.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~295s 2024-08-21T23:30:36.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~295s 2024-08-21T23:30:36.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~295s 2024-08-21T23:30:36.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~295s 2024-08-21T23:30:36.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~295s 2024-08-21T23:30:36.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~295s 2024-08-21T23:30:36.553 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~295s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.0 is failed for ~300s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.1 is failed for ~300s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.osd.2 is failed for ~300s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mon.a is failed for ~300s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~300s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~300s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.mgr.x is failed for ~300s 2024-08-21T23:30:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-08-21T23:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-22T06:19:39.056+0000 7f64ec12a7c0 0 monclient(hunting): authenticate timed out after 300 2024-08-21T23:31:49.847 INFO:teuthology.orchestra.run.smithi116.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-08-21T23:31:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-21T23:31:49.849 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2024-08-21T23:32:34.813 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ERROR 2024-08-21T23:32:34.814 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.814 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_encrypted_object_sync 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1268, in test_encrypted_object_sync 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 179, in zone_meta_checkpoint 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: period, realm_epoch, num_shards, sync_status = meta_sync_status(zone) 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 133, in meta_sync_status 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: meta_sync_status_json, retcode = zone.cluster.admin(cmd, check_retcode=False, read_only=True) 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw_multisite.py", line 195, in admin 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: r, s = rgwadmin(self.ctx, self.client, args, **kwargs) 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/util/rgw.py", line 34, in rgwadmin 2024-08-21T23:32:34.815 INFO:tasks.rgw_multisite_tests: proc = remote.run( 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: r.execute() 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: self.client.exec_command(self.command) 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: chan = self._transport.open_session(timeout=timeout) 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: return self.open_channel( 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: raise e 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run 2024-08-21T23:32:34.816 INFO:tasks.rgw_multisite_tests: ptype, m = self.packetizer.read_message() 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: header = self.read_all(self.__block_size_in, check_rekey=True) 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: x = self.__socket.recv(n) 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: return self._sock.recv(*args) 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests:ConnectionResetError: [Errno 104] Connection reset by peer 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_bucket_index_log_trim 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.817 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1298, in test_bucket_index_log_trim 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1289, in make_test_bucket 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: bucket = zone.conn.create_bucket(name) 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/s3/connection.py", line 618, in create_bucket 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: response = self.make_request('PUT', bucket_name, headers=headers, 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/s3/connection.py", line 667, in make_request 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: return super(S3Connection, self).make_request( 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/connection.py", line 1070, in make_request 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request, sender, override_num_retries, 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/connection.py", line 1030, in _mexe 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: raise ex 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/connection.py", line 942, in _mexe 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1283, in request 2024-08-21T23:32:34.818 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: self.connect() 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 942, in connect 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/socket.py", line 115, in create_connection 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-08-21T23:32:34.819 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests:ERROR: rgw_multi.tests.test_role_sync 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1348, in test_role_sync 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: roles, zone_role = create_role_per_zone(zonegroup_conns) 2024-08-21T23:32:34.820 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 504, in create_role_per_zone 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: role = zone.create_role("", role_name, policy_document, "") 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw_multi/zone_rados.py", line 130, in create_role 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: return self.iam_conn.create_role(rolename, policy_document, path) 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/iam/connection.py", line 1155, in create_role 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: return self.get_response('CreateRole', params) 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/iam/connection.py", line 85, in get_response 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: response = self.make_request(action, params, path, verb) 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/connection.py", line 1116, in make_request 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: return self._mexe(http_request) 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/connection.py", line 1030, in _mexe 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: raise ex 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/boto/connection.py", line 942, in _mexe 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: connection.request(request.method, request.path, 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1283, in request 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: self._send_request(method, url, body, headers, encode_chunked) 2024-08-21T23:32:34.821 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: self.endheaders(body, encode_chunked=encode_chunked) 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: self._send_output(message_body, encode_chunked=encode_chunked) 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: self.send(msg) 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 976, in send 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: self.connect() 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.10/http/client.py", line 942, in connect 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: self.sock = self._create_connection( 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/socket.py", line 115, in create_connection 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: sock.connect(sa) 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 590, in connect 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: self._internal_connect(address) 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests: raise _SocketError(err, strerror(err)) 2024-08-21T23:32:34.822 INFO:tasks.rgw_multisite_tests:ConnectionRefusedError: [Errno 111] Connection refused 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_remove 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 577, in test_bucket_remove 2024-08-21T23:32:34.823 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_object_sync 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 614, in test_object_sync 2024-08-21T23:32:34.824 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_object_delete 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 637, in test_object_delete 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.825 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_versioned_object_incremental_sync 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 674, in test_versioned_object_incremental_sync 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.826 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_concurrent_versioned_object_incremental_sync 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 730, in test_concurrent_versioned_object_incremental_sync 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.827 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_version_suspended_incremental_sync 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 757, in test_version_suspended_incremental_sync 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.828 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_delete_marker_full_sync 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 793, in test_delete_marker_full_sync 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.829 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_suspended_delete_marker_full_sync 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 817, in test_suspended_delete_marker_full_sync 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.830 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 859, in test_bucket_delete_notempty 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.831 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_datalog_autotrim 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 977, in test_datalog_autotrim 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.832 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_zone_redirect 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1020, in test_multi_zone_redirect 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.833 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_zonegroup_remove 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1049, in test_zonegroup_remove 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.834 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1131, in test_bucket_sync_disable 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.835 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1154, in test_bucket_sync_enable_right_after_disable 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.836 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_sync_disable_enable 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1188, in test_bucket_sync_disable_enable 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 202, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 191, in zone_meta_checkpoint 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=a2 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multipart_object_sync 2024-08-21T23:32:34.837 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1232, in test_multipart_object_sync 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 392, in zonegroup_bucket_checkpoint 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: target_conn.check_bucket_eq(source_conn, bucket_name) 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw_multi/zone_rados.py", line 78, in check_bucket_eq 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: assert False 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests:====================================================================== 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_creation_time 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.838 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/nose/case.py", line 170, in runTest 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 1337, in test_bucket_creation_time 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 197, in zonegroup_meta_checkpoint 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: master_status = meta_master_log_status(meta_master_zone) 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/../src/test/rgw/rgw_multi/tests.py", line 143, in meta_master_log_status 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: mdlog_status_json, retcode = master_zone.cluster.admin(cmd, read_only=True) 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw_multisite.py", line 197, in admin 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: assert r == 0 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests:AssertionError 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2024-08-21T23:32:34.839 INFO:tasks.rgw_multisite_tests:Ran 29 tests in 5374.357s 2024-08-21T23:32:34.840 INFO:tasks.rgw_multisite_tests: 2024-08-21T23:32:34.840 INFO:tasks.rgw_multisite_tests:FAILED (errors=3, failures=17) 2024-08-21T23:32:34.840 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw_multisite_tests.py", line 93, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-08-21T23:32:35.086 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7e7c0883b67e48a783cb4a06fb54e803 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw_multisite_tests.py", line 93, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2024-08-21T23:32:35.087 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2024-08-21T23:32:35.096 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2024-08-21T23:32:35.176 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-08-21T23:32:35.238 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2024-08-21T23:32:35.238 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:32:35.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:32:35.239 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" 2024-08-21T23:32:35.248 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-21T23:32:35.315 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2024-08-21T23:32:35.316 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2024-08-21T23:32:35.326 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-08-21T23:32:35.328 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2024-08-21T23:32:35.351 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-21T23:32:35.351 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-21T23:32:35.351 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2024-08-21T23:32:35.351 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:32:35.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:32:35.352 ERROR:teuthology.misc:Saw exception from c2.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0' 2024-08-21T23:32:35.370 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2024-08-21T23:32:35.371 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:32:35.371 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:32:35.371 ERROR:teuthology.misc:Saw exception from c2.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1' 2024-08-21T23:32:35.389 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2024-08-21T23:32:35.389 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:32:35.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:32:35.389 ERROR:teuthology.misc:Saw exception from c2.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' 2024-08-21T23:32:35.408 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-21T23:32:35.408 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2024-08-21T23:32:35.408 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:32:35.408 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:32:35.408 ERROR:teuthology.misc:Saw exception from c2.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1441, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1166, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x' 2024-08-21T23:32:35.430 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-21T23:32:35.431 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2024-08-21T23:32:35.431 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:32:35.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:32:35.431 ERROR:teuthology.misc:Saw exception from c2.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1441, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1166, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1441, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1166, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a' 2024-08-21T23:32:35.457 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-21T23:32:35.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-08-21T23:32:35.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-21T23:32:35.777 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-08-21T23:32:36.107 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-08-21T23:32:36.108 DEBUG:teuthology.orchestra.run.smithi178:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-08-21T23:32:36.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~0s 2024-08-21T23:32:36.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~0s 2024-08-21T23:32:36.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~0s 2024-08-21T23:32:36.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~0s 2024-08-21T23:32:36.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2024-08-21T23:32:36.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2024-08-21T23:32:36.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~0s 2024-08-21T23:32:36.231 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T23:32:36.231 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2024-08-21T23:32:36.397 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T23:32:36.397 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2024-08-21T23:32:36.535 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi178.front.sepia.ceph.com 2024-08-21T23:32:36.536 DEBUG:teuthology.orchestra.run.smithi178:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2024-08-21T23:32:36.690 INFO:tasks.ceph:Archiving mon data... 2024-08-21T23:32:36.692 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274/data/c2.mon.a.tgz 2024-08-21T23:32:36.692 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-08-21T23:32:36.707 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.Ai3Oj7P4DX 2024-08-21T23:32:36.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.Ai3Oj7P4DX 2024-08-21T23:32:36.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.Ai3Oj7P4DX 2024-08-21T23:32:36.983 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.Ai3Oj7P4DX is 400KB 2024-08-21T23:32:37.054 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.Ai3Oj7P4DX 2024-08-21T23:32:37.070 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-21T23:32:37.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-08-21T23:32:37.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2024-08-21T23:32:37.203 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1890, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1104, in cluster yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1441, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1166, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi178 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a' 2024-08-21T23:32:37.214 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-21T23:32:37.224 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2024-08-21T23:32:37.224 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2024-08-21T23:32:41.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~5s 2024-08-21T23:32:41.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~5s 2024-08-21T23:32:41.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~5s 2024-08-21T23:32:41.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~5s 2024-08-21T23:32:41.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~5s 2024-08-21T23:32:41.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~5s 2024-08-21T23:32:41.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~5s 2024-08-21T23:32:46.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~10s 2024-08-21T23:32:46.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~10s 2024-08-21T23:32:46.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~10s 2024-08-21T23:32:46.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~10s 2024-08-21T23:32:46.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~10s 2024-08-21T23:32:46.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~10s 2024-08-21T23:32:46.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~10s 2024-08-21T23:32:51.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~15s 2024-08-21T23:32:51.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~15s 2024-08-21T23:32:51.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~15s 2024-08-21T23:32:51.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~15s 2024-08-21T23:32:51.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~15s 2024-08-21T23:32:51.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~15s 2024-08-21T23:32:51.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~15s 2024-08-21T23:32:56.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~20s 2024-08-21T23:32:56.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~20s 2024-08-21T23:32:56.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~20s 2024-08-21T23:32:56.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~20s 2024-08-21T23:32:56.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~20s 2024-08-21T23:32:56.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~20s 2024-08-21T23:32:56.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~20s 2024-08-21T23:33:01.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~25s 2024-08-21T23:33:01.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~25s 2024-08-21T23:33:01.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~25s 2024-08-21T23:33:01.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~25s 2024-08-21T23:33:01.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~25s 2024-08-21T23:33:01.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~25s 2024-08-21T23:33:01.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~25s 2024-08-21T23:33:06.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~30s 2024-08-21T23:33:06.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~30s 2024-08-21T23:33:06.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~30s 2024-08-21T23:33:06.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~30s 2024-08-21T23:33:06.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~30s 2024-08-21T23:33:06.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~30s 2024-08-21T23:33:06.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~30s 2024-08-21T23:33:11.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~35s 2024-08-21T23:33:11.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~35s 2024-08-21T23:33:11.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~35s 2024-08-21T23:33:11.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~35s 2024-08-21T23:33:11.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~35s 2024-08-21T23:33:11.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~35s 2024-08-21T23:33:11.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~35s 2024-08-21T23:33:16.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~40s 2024-08-21T23:33:16.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~40s 2024-08-21T23:33:16.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~40s 2024-08-21T23:33:16.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~40s 2024-08-21T23:33:16.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~40s 2024-08-21T23:33:16.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~40s 2024-08-21T23:33:16.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~40s 2024-08-21T23:33:21.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~45s 2024-08-21T23:33:21.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~45s 2024-08-21T23:33:21.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~45s 2024-08-21T23:33:21.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~45s 2024-08-21T23:33:21.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2024-08-21T23:33:21.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~45s 2024-08-21T23:33:21.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~45s 2024-08-21T23:33:26.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~50s 2024-08-21T23:33:26.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~50s 2024-08-21T23:33:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~50s 2024-08-21T23:33:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~50s 2024-08-21T23:33:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2024-08-21T23:33:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~50s 2024-08-21T23:33:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~50s 2024-08-21T23:33:31.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~55s 2024-08-21T23:33:31.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~55s 2024-08-21T23:33:31.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~55s 2024-08-21T23:33:31.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~55s 2024-08-21T23:33:31.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~55s 2024-08-21T23:33:31.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~55s 2024-08-21T23:33:31.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~55s 2024-08-21T23:33:36.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~60s 2024-08-21T23:33:36.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~60s 2024-08-21T23:33:36.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~60s 2024-08-21T23:33:36.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~60s 2024-08-21T23:33:36.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~60s 2024-08-21T23:33:36.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~60s 2024-08-21T23:33:36.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~60s 2024-08-21T23:33:41.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~65s 2024-08-21T23:33:41.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~65s 2024-08-21T23:33:41.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~65s 2024-08-21T23:33:41.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~65s 2024-08-21T23:33:41.182 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~65s 2024-08-21T23:33:41.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~65s 2024-08-21T23:33:41.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~65s 2024-08-21T23:33:46.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~70s 2024-08-21T23:33:46.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~70s 2024-08-21T23:33:46.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~70s 2024-08-21T23:33:46.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~70s 2024-08-21T23:33:46.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~70s 2024-08-21T23:33:46.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~70s 2024-08-21T23:33:46.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~70s 2024-08-21T23:33:51.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~75s 2024-08-21T23:33:51.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~75s 2024-08-21T23:33:51.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~75s 2024-08-21T23:33:51.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~75s 2024-08-21T23:33:51.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~75s 2024-08-21T23:33:51.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~75s 2024-08-21T23:33:51.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~75s 2024-08-21T23:33:56.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~80s 2024-08-21T23:33:56.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~80s 2024-08-21T23:33:56.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~80s 2024-08-21T23:33:56.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~80s 2024-08-21T23:33:56.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~80s 2024-08-21T23:33:56.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~80s 2024-08-21T23:33:56.190 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~80s 2024-08-21T23:34:01.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~85s 2024-08-21T23:34:01.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~85s 2024-08-21T23:34:01.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~85s 2024-08-21T23:34:01.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~85s 2024-08-21T23:34:01.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~85s 2024-08-21T23:34:01.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~85s 2024-08-21T23:34:01.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~85s 2024-08-21T23:34:06.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~90s 2024-08-21T23:34:06.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~90s 2024-08-21T23:34:06.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~90s 2024-08-21T23:34:06.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~90s 2024-08-21T23:34:06.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2024-08-21T23:34:06.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~90s 2024-08-21T23:34:06.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~90s 2024-08-21T23:34:11.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~95s 2024-08-21T23:34:11.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~95s 2024-08-21T23:34:11.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~95s 2024-08-21T23:34:11.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~95s 2024-08-21T23:34:11.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2024-08-21T23:34:11.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~95s 2024-08-21T23:34:11.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~95s 2024-08-21T23:34:16.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~100s 2024-08-21T23:34:16.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~100s 2024-08-21T23:34:16.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~100s 2024-08-21T23:34:16.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~100s 2024-08-21T23:34:16.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~100s 2024-08-21T23:34:16.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~100s 2024-08-21T23:34:16.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~100s 2024-08-21T23:34:21.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~105s 2024-08-21T23:34:21.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~105s 2024-08-21T23:34:21.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~105s 2024-08-21T23:34:21.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~105s 2024-08-21T23:34:21.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~105s 2024-08-21T23:34:21.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~105s 2024-08-21T23:34:21.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~105s 2024-08-21T23:34:26.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~110s 2024-08-21T23:34:26.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~110s 2024-08-21T23:34:26.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~110s 2024-08-21T23:34:26.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~110s 2024-08-21T23:34:26.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~110s 2024-08-21T23:34:26.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~110s 2024-08-21T23:34:26.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~110s 2024-08-21T23:34:31.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~115s 2024-08-21T23:34:31.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~115s 2024-08-21T23:34:31.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~115s 2024-08-21T23:34:31.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~115s 2024-08-21T23:34:31.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~115s 2024-08-21T23:34:31.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~115s 2024-08-21T23:34:31.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~115s 2024-08-21T23:34:36.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~120s 2024-08-21T23:34:36.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~120s 2024-08-21T23:34:36.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~120s 2024-08-21T23:34:36.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~120s 2024-08-21T23:34:36.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~120s 2024-08-21T23:34:36.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~120s 2024-08-21T23:34:36.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~120s 2024-08-21T23:34:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-08-21T23:34:37.270 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json' 2024-08-21T23:34:37.287 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-21T23:34:37.287 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-21T23:34:37.288 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2024-08-21T23:34:37.288 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:34:37.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:34:37.293 ERROR:teuthology.misc:Saw exception from c1.osd.0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0' 2024-08-21T23:34:37.310 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2024-08-21T23:34:37.310 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:34:37.310 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:34:37.311 ERROR:teuthology.misc:Saw exception from c1.osd.1 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1' 2024-08-21T23:34:37.326 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2024-08-21T23:34:37.326 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:34:37.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:34:37.327 ERROR:teuthology.misc:Saw exception from c1.osd.2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' 2024-08-21T23:34:37.342 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-21T23:34:37.342 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2024-08-21T23:34:37.342 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:34:37.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:34:37.343 ERROR:teuthology.misc:Saw exception from c1.mgr.x Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1441, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1166, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x' 2024-08-21T23:34:37.365 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-21T23:34:37.366 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2024-08-21T23:34:37.366 INFO:teuthology.orchestra.run:waiting for 300 2024-08-21T23:34:37.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-21T23:34:37.366 ERROR:teuthology.misc:Saw exception from c1.mon.a Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1895, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 452, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 299, in create_pools yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/rgw.py", line 232, in start_rgw ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1899, in task ctx.managers[config['cluster']].stop_pg_num_changes() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2328, in stop_pg_num_changes osd_dump = self.get_osd_dump_json() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 2562, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1645, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph_manager.py", line 1636, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1441, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1166, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1439, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_b5e746614a5ba6992e5be76bf3df447c931f3748/qa/tasks/ceph.py", line 1441, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1166, in stop_daemons_of_type raise exc File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/misc.py", line 1159, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4d4c5e30374204f3b32f867e61bb4e1a3880f672/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi116 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a' 2024-08-21T23:34:37.387 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-21T23:34:37.388 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-08-21T23:34:37.494 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T23:34:37.494 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2024-08-21T23:34:37.603 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T23:34:37.604 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2024-08-21T23:34:37.697 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-21T23:34:37.697 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2024-08-21T23:34:37.773 INFO:tasks.ceph:Archiving mon data... 2024-08-21T23:34:37.774 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274/data/c1.mon.a.tgz 2024-08-21T23:34:37.774 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-08-21T23:34:37.789 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.hGQjNJQ8NV 2024-08-21T23:34:37.790 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.hGQjNJQ8NV 2024-08-21T23:34:37.992 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.hGQjNJQ8NV 2024-08-21T23:34:38.101 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.hGQjNJQ8NV is 518KB 2024-08-21T23:34:38.202 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.hGQjNJQ8NV 2024-08-21T23:34:38.217 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-21T23:34:38.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-08-21T23:34:38.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2024-08-21T23:34:38.291 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-08-21T23:34:38.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-08-21T23:34:38.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-08-21T23:34:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-08-21T23:34:38.386 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-08-21T23:34:38.391 INFO:tasks.ceph:Archiving crash dumps... 2024-08-21T23:34:38.393 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274/remote/smithi116/crash 2024-08-21T23:34:38.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-21T23:34:38.424 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-08-21_20:32:34-rgw-wip-yuri7-testing-2024-08-20-1105-quincy-distro-default-smithi/7866274/remote/smithi178/crash 2024-08-21T23:34:38.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-21T23:34:38.452 INFO:tasks.ceph:Compressing logs... 2024-08-21T23:34:38.452 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-21T23:34:38.469 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-21T23:34:38.499 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.39029.log 2024-08-21T23:34:38.499 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.44837.log 2024-08-21T23:34:38.571 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.24021.log 2024-08-21T23:34:38.571 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23560.log 2024-08-21T23:34:38.571 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.27277.log 2024-08-21T23:34:38.571 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.24021.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.24021.log.gz 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23560.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.27053.log 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23560.log.gz 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.27277.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23979.log 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.27277.log.gz 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.27053.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.25865.log 0.0% 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr: -- replaced with /var/log/ceph/c2-client.0.27053.log.gz 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23979.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23227.log 2024-08-21T23:34:38.572 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23979.log.gz 2024-08-21T23:34:38.573 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.25865.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.28556.log 2024-08-21T23:34:38.573 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.25865.log.gz 2024-08-21T23:34:38.573 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23227.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23433.log 2024-08-21T23:34:38.573 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23227.log.gz 2024-08-21T23:34:38.573 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.28556.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.27382.log 2024-08-21T23:34:38.573 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.28556.log.gz 2024-08-21T23:34:38.573 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23433.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.2.log 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23433.log.gz 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.27382.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.25948.log 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.27300.log 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.25948.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23857.log 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.27300.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.24101.log 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.27300.log.gz 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23857.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23185.log 2024-08-21T23:34:38.574 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23857.log.gz 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.24101.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.28280.log 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.24101.log.gz 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23185.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.20053.log 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23185.log.gz 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.28280.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23647.log 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.28280.log.gz 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.20053.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.24061.log 2024-08-21T23:34:38.575 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.20053.log.gz 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23647.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.19874.log 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23647.log.gz 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.24061.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23687.log 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.24061.log.gz 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.19874.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23773.log 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.19874.log.gz 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23687.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23815.log 2024-08-21T23:34:38.576 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23687.log.gz 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23773.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.24141.log 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23773.log.gz 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23815.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23521.log 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23815.log.gz 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.24141.log: gzip -5 --verbose -- /var/log/ceph/c2-mon.a.log 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.24141.log.gz 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23521.log: gzip -5 --verbose -- /var/log/ceph/c2.audit.log 2024-08-21T23:34:38.577 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23521.log.gz 2024-08-21T23:34:38.578 INFO:teuthology.orchestra.run.smithi178.stderr: 19.7% -- replaced with /var/log/ceph/c2-client.0.27382.log.gz 2024-08-21T23:34:38.578 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.0.log 2024-08-21T23:34:38.578 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2.audit.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23049.log 2024-08-21T23:34:38.578 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.0.log 2024-08-21T23:34:38.578 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23049.log: gzip -5 --verbose -- /var/log/ceph/c2-osd.1.log 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23049.log.gz 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/rgw.c2.client.0.log: gzip -5 --verbose -- /var/log/ceph/c2.log 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.27136.log 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.19954.log 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.0.24204.log 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.27136.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.27136.log.gz 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.19954.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.22966.log 2024-08-21T23:34:38.579 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.19954.log.gz 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.24204.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26171.log 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.22966.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.24181.log 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.22966.log.gz 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.26171.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23897.log 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26171.log.gz 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.24181.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23143.log 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.24181.log.gz 2024-08-21T23:34:38.580 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23897.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23434.log 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23897.log.gz 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23143.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23308.log 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23143.log.gz 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23434.log: gzip -5 --verbose -- /var/log/ceph/c2-client.1.24383.log 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23434.log.gz 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23308.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.28372.log 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23308.log.gz 2024-08-21T23:34:38.581 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.1.24383.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.26129.log 2024-08-21T23:34:38.582 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.28372.log: gzip -5 --verbose -- /var/log/ceph/c2-client.1.24475.log 2024-08-21T23:34:38.582 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.28372.log.gz 2024-08-21T23:34:38.582 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.26129.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23435.log 2024-08-21T23:34:38.582 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.26129.log.gz 2024-08-21T23:34:38.582 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.1.24475.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.19716.log 2024-08-21T23:34:38.582 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23435.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23561.log 2024-08-21T23:34:38.582 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23435.log.gz 2024-08-21T23:34:38.583 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.19716.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23309.log 2024-08-21T23:34:38.583 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.19716.log.gz 2024-08-21T23:34:38.583 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23561.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.27235.log 2024-08-21T23:34:38.583 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23561.log.gz 2024-08-21T23:34:38.583 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23309.log: gzip -5 --verbose -- /var/log/ceph/c2-mgr.x.log 2024-08-21T23:34:38.586 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23309.log.gz 2024-08-21T23:34:38.586 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.27235.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23024.log 2024-08-21T23:34:38.586 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.27235.log.gz 2024-08-21T23:34:38.586 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.24301.log 2024-08-21T23:34:38.586 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23024.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23307.log 2024-08-21T23:34:38.586 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23024.log.gz 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.24301.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.26030.log 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23307.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.20121.log 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23307.log.gz 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/c2-client.admin.19914.log 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.26030.log: 0.0% -- replaced with /var/log/ceph/c2-client.0.26030.log.gz 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.20121.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.26276.log 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.20121.log.gz 2024-08-21T23:34:38.587 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.19914.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.19914.log.gz 2024-08-21T23:34:38.588 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.c2.client.1.log 2024-08-21T23:34:38.588 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.26276.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.26194.log 2024-08-21T23:34:38.588 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/rgw.c2.client.1.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23686.log 2024-08-21T23:34:38.588 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.26194.log: gzip -5 --verbose -- /var/log/ceph/c2-client.0.28464.log 2024-08-21T23:34:38.588 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.26194.log.gz 2024-08-21T23:34:38.588 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23686.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.16426.log 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23686.log.gz 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.0.28464.log: gzip -5 --verbose -- /var/log/ceph/c2-client.admin.23939.log 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.28464.log.gz 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/ceph.tmp-client.admin.16426.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.16426.log.gz 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c2-client.admin.23939.log: 0.0% -- replaced with /var/log/ceph/c2-client.admin.23939.log.gz 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr: 88.3% -- replaced with /var/log/ceph/c2.log.gz 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr: 77.0% -- replaced with /var/log/ceph/c2-client.0.24301.log.gz 2024-08-21T23:34:38.589 INFO:teuthology.orchestra.run.smithi178.stderr: 84.7% -- replaced with /var/log/ceph/c2-client.1.24383.log.gz 2024-08-21T23:34:38.590 INFO:teuthology.orchestra.run.smithi178.stderr: 76.8% -- replaced with /var/log/ceph/c2-client.1.24475.log.gz 2024-08-21T23:34:38.592 INFO:teuthology.orchestra.run.smithi178.stderr: 0.0% -- replaced with /var/log/ceph/c2-client.0.25948.log.gz 2024-08-21T23:34:38.592 INFO:teuthology.orchestra.run.smithi178.stderr: 18.5% -- replaced with /var/log/ceph/c2-client.0.26276.log.gz 2024-08-21T23:34:38.719 INFO:teuthology.orchestra.run.smithi178.stderr: 84.3% -- replaced with /var/log/ceph/c2-client.0.24204.log.gz 2024-08-21T23:34:38.719 INFO:teuthology.orchestra.run.smithi178.stderr: 90.0% -- replaced with /var/log/ceph/c2.audit.log.gz 2024-08-21T23:34:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39029.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56642.log 2024-08-21T23:34:38.896 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.39029.log.gz 2024-08-21T23:34:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44837.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45389.log 2024-08-21T23:34:38.896 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44837.log.gz 2024-08-21T23:34:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56642.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77126.log 2024-08-21T23:34:38.896 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56642.log.gz 2024-08-21T23:34:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45389.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76109.log 2024-08-21T23:34:38.896 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45389.log.gz 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77126.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81828.log 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77126.log.gz 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76109.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65683.log 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76109.log.gz 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81828.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53510.log 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81828.log.gz 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65683.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68543.log 2024-08-21T23:34:38.897 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65683.log.gz 2024-08-21T23:34:38.898 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53510.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88993.log 2024-08-21T23:34:38.898 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53510.log.gz 2024-08-21T23:34:38.898 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68543.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110422.log 2024-08-21T23:34:38.898 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68543.log.gz 2024-08-21T23:34:38.898 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88993.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103584.log 2024-08-21T23:34:38.898 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88993.log.gz 2024-08-21T23:34:38.898 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110422.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121401.log 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110422.log.gz 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103584.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76386.log 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103584.log.gz 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121401.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87179.log 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121401.log.gz 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76386.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106728.log 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76386.log.gz 2024-08-21T23:34:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87179.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36170.log 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87179.log.gz 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106728.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77861.log 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106728.log.gz 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36170.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90748.log 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36170.log.gz 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77861.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98045.log 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77861.log.gz 2024-08-21T23:34:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90748.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66051.log 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90748.log.gz 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98045.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29535.log 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98045.log.gz 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66051.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103030.log 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66051.log.gz 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29535.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47603.log 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29535.log.gz 2024-08-21T23:34:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103030.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62179.log 2024-08-21T23:34:38.902 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103030.log.gz 2024-08-21T23:34:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47603.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46863.log 2024-08-21T23:34:38.902 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47603.log.gz 2024-08-21T23:34:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62179.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42071.log 2024-08-21T23:34:38.902 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62179.log.gz 2024-08-21T23:34:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46863.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.0.log 2024-08-21T23:34:38.902 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46863.log.gz 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42071.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65869.log 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42071.log.gz 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59137.log 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65869.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65775.log 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65869.log.gz 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59137.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41150.log 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59137.log.gz 2024-08-21T23:34:38.903 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65775.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94255.log 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65775.log.gz 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41150.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.41150.log.gz 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118078.log 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94255.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82104.log 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94255.log.gz 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118078.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89085.log 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118078.log.gz 2024-08-21T23:34:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82104.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100541.log 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82104.log.gz 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89085.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114300.log 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89085.log.gz 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.100541.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23068.log 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100541.log.gz 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114300.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.27044.log 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114300.log.gz 2024-08-21T23:34:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23068.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111437.log 2024-08-21T23:34:38.906 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23068.log.gz 2024-08-21T23:34:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.27044.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93701.log 2024-08-21T23:34:38.906 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.27044.log.gz 2024-08-21T23:34:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111437.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85426.log 2024-08-21T23:34:38.906 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111437.log.gz 2024-08-21T23:34:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93701.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95186.log 2024-08-21T23:34:38.906 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93701.log.gz 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85426.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70665.log 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85426.log.gz 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95186.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67435.log 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95186.log.gz 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70665.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32580.log 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70665.log.gz 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67435.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81183.log 2024-08-21T23:34:38.907 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67435.log.gz 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32580.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41887.log 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32580.log.gz 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81183.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23996.log 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81183.log.gz 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41887.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108667.log 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41887.log.gz 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23996.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32384.log 2024-08-21T23:34:38.908 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23996.log.gz 2024-08-21T23:34:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108667.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64024.log 2024-08-21T23:34:38.909 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108667.log.gz 2024-08-21T23:34:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32384.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c1-client.0.32384.log.gz --verbose 2024-08-21T23:34:38.909 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/c1-client.admin.23435.log 2024-08-21T23:34:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64024.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118354.log 2024-08-21T23:34:38.909 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64024.log.gz 2024-08-21T23:34:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23435.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39306.log 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23435.log.gz 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118354.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41242.log 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118354.log.gz 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39306.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.39306.log.gz -5 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/c1-client.0.63840.log 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41242.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38937.log 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41242.log.gz 2024-08-21T23:34:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63840.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102752.log 2024-08-21T23:34:38.911 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63840.log.gz 2024-08-21T23:34:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38937.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52678.log 2024-08-21T23:34:38.911 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38937.log.gz 2024-08-21T23:34:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102752.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116977.log 2024-08-21T23:34:38.911 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102752.log.gz 2024-08-21T23:34:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52678.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120291.log 2024-08-21T23:34:38.911 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52678.log.gz 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116977.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83303.log 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116977.log.gz 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120291.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117533.log 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120291.log.gz 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83303.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83584.log 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83303.log.gz 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117533.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114852.log 2024-08-21T23:34:38.912 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117533.log.gz 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83584.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69835.log 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83584.log.gz 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114852.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46035.log 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114852.log.gz 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69835.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105157.log 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69835.log.gz 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46035.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23914.log 2024-08-21T23:34:38.913 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46035.log.gz 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105157.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35434.log 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105157.log.gz 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23914.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43640.log 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23914.log.gz 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35434.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102844.log 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35434.log.gz 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43640.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70389.log 2024-08-21T23:34:38.914 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43640.log.gz 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102844.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44008.log 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102844.log.gz 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70389.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66235.log 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70389.log.gz 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44008.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31292.log 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44008.log.gz 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66235.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34606.log 2024-08-21T23:34:38.915 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66235.log.gz 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31292.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83395.log 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31292.log.gz 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34606.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82012.log 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34606.log.gz 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83395.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63011.log 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83395.log.gz 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82012.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.82012.log.gz 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.103770.log 2024-08-21T23:34:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65039.log 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63011.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.63011.log.gz 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103770.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97585.log 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103770.log.gz 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65039.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.28799.log 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65039.log.gz 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97585.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116331.log 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97585.log.gz 2024-08-21T23:34:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.28799.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33686.log 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.28799.log.gz 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116331.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122229.log 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116331.log.gz 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33686.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.33686.log.gz 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.30179.log 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.122229.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96201.log 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122229.log.gz 2024-08-21T23:34:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30179.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/c1-client.0.30179.log.gz /var/log/ceph/c1-client.0.91854.log 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96201.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79153.log 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96201.log.gz 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91854.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97769.log 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91854.log.gz 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79153.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104694.log 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79153.log.gz 2024-08-21T23:34:38.919 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97769.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.30455.log 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97769.log.gz 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104694.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.19951.log 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104694.log.gz 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30455.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113746.log 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30455.log.gz 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.19951.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.26020.log 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.19951.log.gz 2024-08-21T23:34:38.920 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113746.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104048.log 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113746.log.gz 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.26020.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117349.log 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.26020.log.gz 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104048.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79985.log 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104048.log.gz 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117349.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59044.log 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117349.log.gz 2024-08-21T23:34:38.921 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79985.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59413.log 2024-08-21T23:34:38.922 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79985.log.gz 2024-08-21T23:34:38.922 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59044.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107467.log 2024-08-21T23:34:38.922 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59044.log.gz 2024-08-21T23:34:38.922 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59413.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.26266.log 2024-08-21T23:34:38.922 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59413.log.gz 2024-08-21T23:34:38.922 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107467.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63472.log 2024-08-21T23:34:38.922 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107467.log.gz 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.26266.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68727.log 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63472.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111253.log 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63472.log.gz 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68727.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69282.log 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68727.log.gz 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111253.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84874.log 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111253.log.gz 2024-08-21T23:34:38.923 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69282.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36078.log 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69282.log.gz 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84874.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84228.log 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84874.log.gz 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr: 19.1% -- replaced with /var/log/ceph/c1-client.0.26266.log.gz 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36078.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61073.log 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36078.log.gz 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84228.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108019.log 2024-08-21T23:34:38.924 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84228.log.gz 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61073.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35250.log 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61073.log.gz 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108019.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73988.log 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108019.log.gz 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35250.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51383.log 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35250.log.gz 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73988.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39214.log 2024-08-21T23:34:38.925 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73988.log.gz 2024-08-21T23:34:38.926 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51383.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69098.log 0.0% 2024-08-21T23:34:38.926 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.51383.log.gz 2024-08-21T23:34:38.926 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39214.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38015.log 2024-08-21T23:34:38.926 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.39214.log.gz 2024-08-21T23:34:38.926 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69098.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35710.log 2024-08-21T23:34:38.926 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69098.log.gz 2024-08-21T23:34:38.926 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38015.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80170.log 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38015.log.gz 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35710.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77492.log 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35710.log.gz 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80170.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43361.log 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80170.log.gz 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77492.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42992.log 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77492.log.gz 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43361.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34146.log 2024-08-21T23:34:38.927 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43361.log.gz 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42992.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37645.log 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42992.log.gz 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34146.log: gzip -5 --verbose -- /var/log/ceph/c1-mgr.x.log 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34146.log.gz 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37645.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121953.log 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37645.log.gz 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63103.log 2024-08-21T23:34:38.928 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121953.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79615.log 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121953.log.gz 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63103.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80723.log 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63103.log.gz 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79615.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93425.log 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79615.log.gz 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80723.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38661.log 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80723.log.gz 2024-08-21T23:34:38.929 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93425.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86440.log 2024-08-21T23:34:38.930 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93425.log.gz 2024-08-21T23:34:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38661.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45943.log 2024-08-21T23:34:38.930 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38661.log.gz 2024-08-21T23:34:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86440.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35802.log 2024-08-21T23:34:38.930 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86440.log.gz 2024-08-21T23:34:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45943.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106636.log 2024-08-21T23:34:38.930 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45943.log.gz 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35802.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53233.log 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35802.log.gz 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106636.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67622.log 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106636.log.gz 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53233.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75279.log 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53233.log.gz 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67622.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77307.log 2024-08-21T23:34:38.931 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67622.log.gz 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75279.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101738.log 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75279.log.gz 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77307.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98137.log 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77307.log.gz 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101738.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121493.log 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101738.log.gz 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98137.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40598.log 2024-08-21T23:34:38.932 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98137.log.gz 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121493.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101462.log 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121493.log.gz 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40598.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116053.log 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40598.log.gz 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101462.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.101462.log.gz 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.51569.log 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116053.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84966.log 2024-08-21T23:34:38.933 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116053.log.gz 2024-08-21T23:34:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51569.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99060.log 2024-08-21T23:34:38.934 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51569.log.gz 2024-08-21T23:34:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84966.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86995.log 2024-08-21T23:34:38.934 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84966.log.gz 2024-08-21T23:34:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99060.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.99060.log.gz 2024-08-21T23:34:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109869.log 2024-08-21T23:34:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86995.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60702.log 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86995.log.gz 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109869.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107375.log 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109869.log.gz 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60702.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69650.log 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60702.log.gz 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107375.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78230.log 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107375.log.gz 2024-08-21T23:34:38.935 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69650.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73247.log 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69650.log.gz 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78230.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66606.log 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78230.log.gz 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73247.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113654.log 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73247.log.gz 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66606.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.66606.log.gz 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.36631.log 2024-08-21T23:34:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113654.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29995.log 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113654.log.gz 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36631.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29627.log 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36631.log.gz 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.108851.log 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29995.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.29995.log.gz 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29627.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78046.log 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29627.log.gz 2024-08-21T23:34:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108851.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37091.log 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108851.log.gz 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78046.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66143.log 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78046.log.gz 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37091.log: gzip -5 --verbose 0.0% -- /var/log/ceph/c1-client.0.67806.log -- replaced with /var/log/ceph/c1-client.0.37091.log.gz 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66143.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61165.log 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66143.log.gz 2024-08-21T23:34:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67806.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.27274.log 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67806.log.gz 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61165.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95739.log 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61165.log.gz 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.27274.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117161.log 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.27274.log.gz 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95739.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50274.log 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95739.log.gz 2024-08-21T23:34:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117161.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103862.log 2024-08-21T23:34:38.940 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117161.log.gz 2024-08-21T23:34:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50274.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/c1-client.0.50274.log.gz /var/log/ceph/c1-client.0.39122.log 2024-08-21T23:34:38.940 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103862.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62734.log 2024-08-21T23:34:38.940 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103862.log.gz 2024-08-21T23:34:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39122.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104324.log 2024-08-21T23:34:38.940 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.39122.log.gz 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62734.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119921.log 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62734.log.gz 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104324.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52492.log 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104324.log.gz 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119921.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.30731.log 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119921.log.gz 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52492.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61811.log 2024-08-21T23:34:38.941 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52492.log.gz 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30731.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113561.log 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30731.log.gz 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61811.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c1-client.0.61811.log.gz /var/log/ceph/c1-client.0.63748.log 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113561.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40966.log 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113561.log.gz 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63748.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104789.log 2024-08-21T23:34:38.942 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63748.log.gz 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40966.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38569.log 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40966.log.gz 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104789.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23310.log 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104789.log.gz 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38569.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.38569.log.gz 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40690.log 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23310.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61441.log 2024-08-21T23:34:38.943 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23310.log.gz 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40690.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113284.log 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40690.log.gz 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/c1-client.0.61441.log: -5 --verbose -- /var/log/ceph/c1-client.0.33962.log 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61441.log.gz 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113284.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29903.log 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113284.log.gz 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33962.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101646.log 2024-08-21T23:34:38.944 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.33962.log.gz 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29903.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45573.log 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29903.log.gz 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101646.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35618.log 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101646.log.gz 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45573.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51753.log 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45573.log.gz 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35618.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108111.log 2024-08-21T23:34:38.945 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35618.log.gz 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51753.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65131.log 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51753.log.gz 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108111.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54432.log 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108111.log.gz 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65131.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.65131.log.gz 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.53972.log 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54432.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87547.log 2024-08-21T23:34:38.946 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54432.log.gz 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53972.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78874.log 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53972.log.gz 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87547.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40782.log 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87547.log.gz 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78874.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75925.log 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78874.log.gz 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40782.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39677.log 2024-08-21T23:34:38.947 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40782.log.gz 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75925.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53880.log 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75925.log.gz 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39677.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65591.log 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.39677.log.gz 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53880.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33317.log 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53880.log.gz 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65591.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89824.log 2024-08-21T23:34:38.948 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65591.log.gz 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95647.log 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33317.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.33317.log.gz 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89824.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116793.log 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89824.log.gz 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95647.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55627.log 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95647.log.gz 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48986.log 2024-08-21T23:34:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116793.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.116793.log.gz 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55627.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58307.log 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55627.log.gz 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48986.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31657.log 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48986.log.gz 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58307.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71035.log 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58307.log.gz 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31657.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58583.log 2024-08-21T23:34:38.950 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31657.log.gz 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71035.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120752.log 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71035.log.gz 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58583.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71404.log 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58583.log.gz 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120752.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120660.log 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120752.log.gz 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71404.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58675.log 2024-08-21T23:34:38.951 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71404.log.gz 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120660.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102106.log 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120660.log.gz 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.25938.log 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58675.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.58675.log.gz 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102106.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59783.log 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102106.log.gz 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.25938.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70481.log 2024-08-21T23:34:38.952 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59783.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99430.log 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59783.log.gz 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70481.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115684.log 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70481.log.gz 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99430.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41795.log 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99430.log.gz 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.38293.log 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115684.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.115684.log.gz 2024-08-21T23:34:38.953 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41795.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70850.log 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41795.log.gz 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38293.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49446.log 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38293.log.gz 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70850.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81735.log 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70850.log.gz 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49446.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40045.log 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49446.log.gz 2024-08-21T23:34:38.954 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81735.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89548.log 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81735.log.gz 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72140.log 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40045.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.40045.log.gz 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89548.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85888.log 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89548.log.gz 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72140.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121677.log 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72140.log.gz 2024-08-21T23:34:38.955 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.25938.log.gz 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85888.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48892.log 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85888.log.gz 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121677.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64116.log 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121677.log.gz 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48892.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60886.log 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48892.log.gz 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64116.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84690.log 2024-08-21T23:34:38.956 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64116.log.gz 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67251.log 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60886.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.60886.log.gz 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84690.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64208.log 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84690.log.gz 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67251.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90471.log 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67251.log.gz 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64208.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.27232.log 2024-08-21T23:34:38.957 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64208.log.gz 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90471.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105249.log 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90471.log.gz 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.27232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.19784.log 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.27232.log.gz 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105249.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100064.log 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105249.log.gz 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.19784.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94621.log 2024-08-21T23:34:38.958 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.19784.log.gz 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.100064.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72603.log 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100064.log.gz 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/c1-client.0.94621.log: -5 --verbose -- /var/log/ceph/c1-client.0.122137.log 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94621.log.gz 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72603.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33778.log 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72603.log.gz 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.122137.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96017.log 2024-08-21T23:34:38.959 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122137.log.gz 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33778.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64576.log 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.33778.log.gz 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96017.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34422.log 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96017.log.gz 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64576.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83211.log 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64576.log.gz 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34422.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.34422.log.gz 2024-08-21T23:34:38.960 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86164.log 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83211.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51015.log 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83211.log.gz 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86164.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111624.log 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86164.log.gz 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51015.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49722.log 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51015.log.gz 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47233.log 2024-08-21T23:34:38.961 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111624.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.111624.log.gz 2024-08-21T23:34:38.962 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49722.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.30087.log 2024-08-21T23:34:38.962 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49722.log.gz 2024-08-21T23:34:38.962 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47233.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121861.log 2024-08-21T23:34:38.962 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47233.log.gz 2024-08-21T23:34:38.962 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30087.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115408.log 2024-08-21T23:34:38.962 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30087.log.gz 2024-08-21T23:34:38.962 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121861.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44468.log 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121861.log.gz 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115408.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98968.log 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115408.log.gz 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44468.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33870.log 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44468.log.gz 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98968.log: gzip -5 --verbose -- /var/log/ceph/c1-osd.1.log 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98968.log.gz 2024-08-21T23:34:38.963 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33870.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54798.log 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.33870.log.gz 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98600.log 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54798.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92871.log 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54798.log.gz 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98600.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105437.log 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98600.log.gz 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86072.log 2024-08-21T23:34:38.964 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92871.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.92871.log.gz 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105437.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.105437.log.gz -5 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/c1-client.0.74635.log 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86072.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45758.log 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86072.log.gz 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74635.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51199.log 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74635.log.gz 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45758.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95925.log 2024-08-21T23:34:38.965 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45758.log.gz 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51199.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50831.log 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51199.log.gz 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95925.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74727.log 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95925.log.gz 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50831.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113838.log 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50831.log.gz 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74727.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66882.log 2024-08-21T23:34:38.966 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74727.log.gz 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113838.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.76846.log 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.113838.log.gz 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66882.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37923.log 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66882.log.gz 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76846.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67898.log 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76846.log.gz 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37923.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46127.log 2024-08-21T23:34:38.967 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37923.log.gz 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.115316.log 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67898.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67898.log.gz 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46127.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.28621.log 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46127.log.gz 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115316.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110609.log 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115316.log.gz 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.28621.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78782.log 2024-08-21T23:34:38.968 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.28621.log.gz 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110609.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47971.log 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110609.log.gz 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78782.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52216.log 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78782.log.gz 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47971.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.47971.log.gz 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.104509.log 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52216.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62550.log 2024-08-21T23:34:38.969 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52216.log.gz 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104509.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86627.log 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104509.log.gz 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62550.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108391.log 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62550.log.gz 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86627.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63195.log 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86627.log.gz 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108391.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73526.log 2024-08-21T23:34:38.970 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108391.log.gz 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63195.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83768.log 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63195.log.gz 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73526.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112916.log 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73526.log.gz 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83768.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.16257.log 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83768.log.gz 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112916.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117717.log 2024-08-21T23:34:38.971 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112916.log.gz 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.tmp-client.admin.16257.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86348.log 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.16257.log.gz 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117717.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.117717.log.gz 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80631.log 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86348.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90841.log 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86348.log.gz 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80631.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61625.log 2024-08-21T23:34:38.972 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80631.log.gz 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90841.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109499.log 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90841.log.gz 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61625.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60059.log 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61625.log.gz 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109499.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c1-client.0.109499.log.gz 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/c1-client.0.112454.log 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60059.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84044.log 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60059.log.gz 2024-08-21T23:34:38.973 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112454.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37740.log 2024-08-21T23:34:38.974 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112454.log.gz 2024-08-21T23:34:38.974 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.112546.log 2024-08-21T23:34:38.974 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84044.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84044.log.gz 2024-08-21T23:34:38.974 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37740.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.37740.log.gz 2024-08-21T23:34:38.974 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.100419.log 2024-08-21T23:34:38.974 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112546.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23522.log 2024-08-21T23:34:38.974 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112546.log.gz 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.100419.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119185.log 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23522.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101830.log 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23522.log.gz 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119185.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.123232.log 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119185.log.gz 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101830.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60243.log 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101830.log.gz 2024-08-21T23:34:38.975 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.123232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85058.log 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr: 6.1% -- replaced with /var/log/ceph/c1-client.0.123232.log.gz 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60243.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65959.log 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60243.log.gz 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85058.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108759.log 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85058.log.gz 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr: 5.8% -- replaced with /var/log/ceph/c1-client.0.100419.log.gz 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65959.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80907.log 2024-08-21T23:34:38.976 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65959.log.gz 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108759.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.108759.log.gz 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.119645.log 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80907.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114022.log 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80907.log.gz 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119645.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.119645.log.gz 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.82472.log 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114022.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113930.log 2024-08-21T23:34:38.977 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114022.log.gz 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.100146.log 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82472.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.82472.log.gz 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113930.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73896.log 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113930.log.gz 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.100146.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93333.log 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100146.log.gz 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73896.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.73896.log.gz 2024-08-21T23:34:38.978 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.48708.log 2024-08-21T23:34:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93333.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91209.log 2024-08-21T23:34:38.979 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93333.log.gz 2024-08-21T23:34:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48708.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61995.log 2024-08-21T23:34:38.979 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48708.log.gz 2024-08-21T23:34:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91209.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62366.log 2024-08-21T23:34:38.979 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91209.log.gz 2024-08-21T23:34:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61995.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.61995.log.gz 2024-08-21T23:34:38.980 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.81551.log 2024-08-21T23:34:38.980 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62366.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105898.log 2024-08-21T23:34:38.980 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62366.log.gz 2024-08-21T23:34:38.980 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81551.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46219.log 2024-08-21T23:34:38.980 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81551.log.gz 2024-08-21T23:34:38.980 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105898.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72971.log 2024-08-21T23:34:38.980 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105898.log.gz 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46219.log: gzip -5 --verbose -- /var/log/ceph/c1.audit.log 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46219.log.gz 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72971.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105713.log 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72971.log.gz 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1.audit.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23228.log 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105713.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101186.log 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105713.log.gz 2024-08-21T23:34:38.981 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23228.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67990.log 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23228.log.gz 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101186.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118630.log 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101186.log.gz 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67990.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23012.log 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67990.log.gz 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118630.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43176.log 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118630.log.gz 2024-08-21T23:34:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.35066.log 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23012.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23012.log.gz 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43176.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86811.log 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43176.log.gz 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35066.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57011.log 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35066.log.gz 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86811.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101094.log 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86811.log.gz 2024-08-21T23:34:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57011.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57291.log 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57011.log.gz 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101094.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74911.log 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101094.log.gz 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57291.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107743.log 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57291.log.gz 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74911.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52864.log 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74911.log.gz 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107743.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33594.log 2024-08-21T23:34:38.984 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107743.log.gz 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.41703.log 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52864.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.52864.log.gz 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87823.log 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33594.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.33594.log.gz 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41703.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41335.log 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41703.log.gz 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87823.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31565.log 2024-08-21T23:34:38.985 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87823.log.gz 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41335.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112181.log 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41335.log.gz 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31565.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.31565.log.gz 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69374.log 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112181.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41058.log 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112181.log.gz 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69374.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102936.log 2024-08-21T23:34:38.986 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69374.log.gz 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41058.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119369.log 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41058.log.gz 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102936.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86256.log 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102936.log.gz 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119369.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67343.log 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119369.log.gz 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86256.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96938.log 2024-08-21T23:34:38.987 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86256.log.gz 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.106450.log 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67343.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.67343.log.gz 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96938.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40322.log 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96938.log.gz 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106450.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89456.log 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106450.log.gz 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40322.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111900.log 2024-08-21T23:34:38.988 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40322.log.gz 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89456.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52123.log 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89456.log.gz 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111900.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51107.log 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111900.log.gz 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52123.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.52123.log.gz 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87731.log 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51107.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88377.log 2024-08-21T23:34:38.989 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51107.log.gz 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87731.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.24156.log 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87731.log.gz 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88377.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59599.log 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88377.log.gz 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.24156.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.24156.log.gz 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.105621.log 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59599.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65499.log 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59599.log.gz 2024-08-21T23:34:38.990 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105621.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93517.log 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105621.log.gz 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65499.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29443.log 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65499.log.gz 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93517.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23872.log 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93517.log.gz 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29443.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104973.log 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29443.log.gz 2024-08-21T23:34:38.991 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23872.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116145.log 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23872.log.gz 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104973.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121214.log 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104973.log.gz 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116145.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77399.log 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116145.log.gz 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121214.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23562.log 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121214.log.gz 2024-08-21T23:34:38.992 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77399.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72510.log 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77399.log.gz 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23562.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71128.log 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23562.log.gz 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.114116.log 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72510.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.72510.log.gz 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71128.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39769.log 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71128.log.gz 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114116.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34238.log 2024-08-21T23:34:38.993 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114116.log.gz 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.75003.log 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39769.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39769.log.gz 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34238.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65315.log 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34238.log.gz 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60794.log 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75003.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.75003.log.gz 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65315.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93887.log 2024-08-21T23:34:38.994 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65315.log.gz 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/c1-client.0.60794.log: -5 --verbose -- /var/log/ceph/c1-client.0.110885.log 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60794.log.gz 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/c1-client.0.93887.log: -5 --verbose -- /var/log/ceph/c1-client.0.117989.log 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93887.log.gz 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110885.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79800.log 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110885.log.gz 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117989.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c1-client.0.117989.log.gz --verbose 2024-08-21T23:34:38.995 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/c1-client.0.95278.log 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79800.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76201.log 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79800.log.gz 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95278.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59507.log 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95278.log.gz 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76201.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32488.log 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76201.log.gz 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59507.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.59507.log.gz 2024-08-21T23:34:38.996 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.92687.log 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32488.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103956.log 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32488.log.gz 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92687.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54983.log 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92687.log.gz 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103956.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100913.log 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103956.log.gz 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54983.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69929.log 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54983.log.gz 2024-08-21T23:34:38.997 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.100913.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46771.log 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100913.log.gz 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69929.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66698.log 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69929.log.gz 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46771.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85334.log 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46771.log.gz 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66698.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68266.log 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66698.log.gz 2024-08-21T23:34:38.998 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85334.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.85334.log.gz 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.112638.log 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68266.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69006.log 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68266.log.gz 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112638.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68635.log 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112638.log.gz 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69006.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113100.log 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69006.log.gz 2024-08-21T23:34:38.999 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/c1-client.0.68635.log: --verbose -- /var/log/ceph/c1-client.0.51845.log 2024-08-21T23:34:39.000 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68635.log.gz 2024-08-21T23:34:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113100.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109591.log 2024-08-21T23:34:39.000 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113100.log.gz 2024-08-21T23:34:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51845.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95554.log 2024-08-21T23:34:39.000 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51845.log.gz 2024-08-21T23:34:39.000 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109591.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115777.log 2024-08-21T23:34:39.000 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109591.log.gz 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95554.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80262.log 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95554.log.gz 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115777.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29075.log 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115777.log.gz 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80262.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120476.log 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80262.log.gz 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29075.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108298.log 2024-08-21T23:34:39.001 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29075.log.gz 2024-08-21T23:34:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.64947.log 2024-08-21T23:34:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120476.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.120476.log.gz 2024-08-21T23:34:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108298.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84322.log 2024-08-21T23:34:39.002 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108298.log.gz 2024-08-21T23:34:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64947.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59691.log 2024-08-21T23:34:39.002 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64947.log.gz 2024-08-21T23:34:39.002 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76017.log 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84322.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84322.log.gz 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59691.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80999.log 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59691.log.gz 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76017.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23561.log 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76017.log.gz 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/c1-client.0.80999.log: -- /var/log/ceph/c1-client.0.24179.log 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80999.log.gz 2024-08-21T23:34:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88718.log 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23561.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23561.log.gz 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.24179.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35526.log 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88718.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49168.log 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88718.log.gz 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35526.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37367.log 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35526.log.gz 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49168.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59967.log 2024-08-21T23:34:39.004 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49168.log.gz 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37367.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35894.log 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37367.log.gz 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59967.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80354.log 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59967.log.gz 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35894.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107651.log 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35894.log.gz 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80354.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23434.log 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80354.log.gz 2024-08-21T23:34:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107651.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56734.log 2024-08-21T23:34:39.006 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107651.log.gz 2024-08-21T23:34:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23434.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68174.log 2024-08-21T23:34:39.006 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23434.log.gz 2024-08-21T23:34:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56734.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122045.log 2024-08-21T23:34:39.006 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56734.log.gz 2024-08-21T23:34:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68174.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46403.log 2024-08-21T23:34:39.006 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68174.log.gz 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87916.log 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.122045.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.122045.log.gz 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46403.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80815.log 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46403.log.gz 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87916.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23309.log 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87916.log.gz 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr: 84.4% -- replaced with /var/log/ceph/c1-client.0.24179.log.gz 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80815.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.80815.log.gz 2024-08-21T23:34:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.59875.log 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23309.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109035.log 0.0% 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.admin.23309.log.gz 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59875.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.59875.log.gz 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.117441.log 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109035.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23093.log 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109035.log.gz 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117441.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32948.log 2024-08-21T23:34:39.008 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117441.log.gz 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23093.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.26184.log 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23093.log.gz 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32948.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114392.log 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32948.log.gz 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.26184.log: gzip -5 --verbose -- /var/log/ceph/c1-client.1.24450.log 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.26184.log.gz 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114392.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71863.log 2024-08-21T23:34:39.009 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114392.log.gz 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.1.24450.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.24076.log 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71863.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50182.log 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71863.log.gz 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.24076.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114944.log 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.24076.log.gz 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr: 76.8% -- replaced with /var/log/ceph/c1-client.1.24450.log.gz 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50182.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44652.log 2024-08-21T23:34:39.010 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50182.log.gz 2024-08-21T23:34:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114944.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97953.log 2024-08-21T23:34:39.011 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114944.log.gz 2024-08-21T23:34:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/c1-client.0.44652.log: --verbose -- /var/log/ceph/c1-client.0.70757.log 2024-08-21T23:34:39.011 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44652.log.gz 2024-08-21T23:34:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97953.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89640.log 2024-08-21T23:34:39.011 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97953.log.gz 2024-08-21T23:34:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70757.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113376.log 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70757.log.gz 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89640.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97401.log 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89640.log.gz 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.98784.log 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113376.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.113376.log.gz 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97401.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.97401.log.gz 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.31101.log 2024-08-21T23:34:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98784.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103677.log 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98784.log.gz 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31101.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48156.log 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31101.log.gz 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103677.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107835.log 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103677.log.gz 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/c1.audit.log.gz 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48156.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114484.log 2024-08-21T23:34:39.013 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48156.log.gz 2024-08-21T23:34:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107835.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88807.log 2024-08-21T23:34:39.014 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107835.log.gz 2024-08-21T23:34:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114484.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85612.log 2024-08-21T23:34:39.014 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114484.log.gz 2024-08-21T23:34:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.30363.log 2024-08-21T23:34:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88807.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.88807.log.gz 2024-08-21T23:34:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85612.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96846.log 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85612.log.gz 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30363.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.30363.log.gz 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.98692.log 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96846.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33409.log 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96846.log.gz 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98692.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91302.log 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98692.log.gz 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/c1-client.0.33409.log: -5 --verbose -- /var/log/ceph/c1-client.0.41611.log 2024-08-21T23:34:39.015 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.33409.log.gz 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91302.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75741.log 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91302.log.gz 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41611.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68082.log 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41611.log.gz 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75741.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102198.log 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75741.log.gz 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68082.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39398.log 2024-08-21T23:34:39.016 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68082.log.gz 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102198.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53049.log 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102198.log.gz 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47879.log 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39398.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39398.log.gz 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53049.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73804.log 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53049.log.gz 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.79245.log 2024-08-21T23:34:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87363.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118538.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.82933.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60981.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.121120.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84136.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83860.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63932.log 2024-08-21T23:34:39.018 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.77215.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118170.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74819.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46957.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.39490.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.48524.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.63656.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.24116.log 2024-08-21T23:34:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81367.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47325.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-osd.2.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47417.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109961.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90933.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.95094.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.117899.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72418.log 2024-08-21T23:34:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.82196.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78966.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.31384.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.97861.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.107191.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83026.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78506.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.49906.log 2024-08-21T23:34:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.32200.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78598.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.87271.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109776.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.34790.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.34698.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47695.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.39861.log 2024-08-21T23:34:39.022 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.110330.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.113468.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.106546.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.33502.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.104601.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50368.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62827.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.77031.log 2024-08-21T23:34:39.023 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.76478.log 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.120384.log 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.120015.log 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.31203.log 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60518.log 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47879.log: /var/log/ceph/c1-client.0.120015.log: /var/log/ceph/c1-client.0.73804.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.47879.log.gz/var/log/ceph/c1-client.0.79245.log: 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120015.log.gz 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0%/var/log/ceph/c1-client.0.87363.log: -- replaced with /var/log/ceph/c1-client.0.79245.log.gz 2024-08-21T23:34:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118538.log: /var/log/ceph/c1-client.0.31203.log: /var/log/ceph/c1-client.0.82933.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.73804.log.gz 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87363.log.gz 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31203.log.gz 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118538.log.gz 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82933.log.gz 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60981.log: /var/log/ceph/c1-client.0.121120.log: /var/log/ceph/c1-client.0.84136.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104417.log 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60981.log.gz 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83860.log: /var/log/ceph/c1-client.0.60518.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.121120.log.gz 2024-08-21T23:34:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63932.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.83860.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84136.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77215.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.63932.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60518.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118170.log: /var/log/ceph/c1-client.0.74819.log: /var/log/ceph/c1-client.0.46957.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.77215.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39490.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.118170.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48524.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.74819.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46957.log.gz 2024-08-21T23:34:39.026 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.39490.log.gz 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63656.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89917.log 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.24116.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.48524.log.gz 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81367.log: /var/log/ceph/c1-client.0.47325.log: /var/log/ceph/c1-osd.2.log: 0.0% -- replaced with /var/log/ceph/c1-client.admin.24116.log.gz 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63656.log.gz 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47417.log: /var/log/ceph/c1-client.0.104417.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.81367.log.gz 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47325.log.gz 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47417.log.gz/var/log/ceph/c1-client.0.109961.log: /var/log/ceph/c1-client.0.90933.log: 2024-08-21T23:34:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95094.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.109961.log.gz 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90933.log.gz 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95094.log.gz 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.90194.log 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104417.log.gz 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117899.log: /var/log/ceph/c1-client.0.72418.log: /var/log/ceph/c1-client.0.82196.log: /var/log/ceph/c1-client.0.78966.log: /var/log/ceph/c1-client.0.31384.log: /var/log/ceph/c1-client.0.97861.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.72418.log.gz 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89917.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.82196.log.gz 0.0% 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.31384.log.gz 2024-08-21T23:34:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50552.log 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78966.log.gz 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0%/var/log/ceph/c1-client.0.107191.log: -- replaced with /var/log/ceph/c1-client.0.89917.log.gz 32.5% -- replaced with /var/log/ceph/c1-client.0.117899.log.gz 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83026.log: 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/c1-client.0.107191.log.gz 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.97861.log.gz 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83026.log.gz 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78506.log: /var/log/ceph/c1-client.0.49906.log: /var/log/ceph/c1-client.0.32200.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66329.log 2024-08-21T23:34:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78598.log: 0.0%/var/log/ceph/c1-client.0.90194.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.78506.log.gz 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78598.log.gz 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.49906.log.gz 0.0% 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.32200.log.gz 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90194.log.gz 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87271.log: /var/log/ceph/c1-client.0.109776.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23648.log 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34790.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.87271.log.gz 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0%/var/log/ceph/c1-client.0.34698.log: /var/log/ceph/c1-client.0.50552.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.34790.log.gz 2024-08-21T23:34:39.030 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.109776.log.gz 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34698.log.gz 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50552.log.gz 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66329.log: /var/log/ceph/c1-client.0.47695.log: 0.0%/var/log/ceph/c1-client.0.39861.log: -- replaced with /var/log/ceph/c1-client.0.66329.log.gz 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.43268.log 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110330.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.47695.log.gz 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113468.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.110330.log.gz 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106546.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39861.log.gz 2024-08-21T23:34:39.031 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33502.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.113468.log.gz 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104601.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.106546.log.gz 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23648.log: 0.0%gzip -5 --verbose -- /var/log/ceph/c1-client.0.120936.log 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50368.log: /var/log/ceph/c1-client.0.62827.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.33502.log.gz 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23648.log.gz 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77031.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.104601.log.gz 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62827.log.gz 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76478.log: -- replaced with /var/log/ceph/c1-client.0.50368.log.gz/var/log/ceph/c1-client.0.120384.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.77031.log.gz 2024-08-21T23:34:39.032 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76478.log.gz 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120384.log.gz 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.101002.log/var/log/ceph/c1-client.0.43268.log: 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43268.log.gz 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120936.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97677.log 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120936.log.gz 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101002.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81091.log 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101002.log.gz 2024-08-21T23:34:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97677.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73711.log 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97677.log.gz 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81091.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91025.log 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81091.log.gz 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73711.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108575.log 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73711.log.gz 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91025.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120844.log 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91025.log.gz 2024-08-21T23:34:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108575.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41427.log 2024-08-21T23:34:39.035 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108575.log.gz 2024-08-21T23:34:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120844.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79058.log 2024-08-21T23:34:39.035 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120844.log.gz 2024-08-21T23:34:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41427.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41519.log 2024-08-21T23:34:39.035 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41427.log.gz 2024-08-21T23:34:39.035 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79058.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107927.log 2024-08-21T23:34:39.035 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79058.log.gz 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41519.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31012.log 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.41519.log.gz 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107927.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.19744.log 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107927.log.gz 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31012.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.28710.log 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31012.log.gz 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.19744.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36446.log 2024-08-21T23:34:39.036 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.19744.log.gz 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.28710.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45021.log 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.28710.log.gz 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36446.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35158.log 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36446.log.gz 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45021.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.45021.log.gz 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.116423.log 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35158.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111069.log 2024-08-21T23:34:39.037 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35158.log.gz 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116423.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94437.log 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116423.log.gz 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111069.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23186.log 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111069.log.gz 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.115129.log 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94437.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.94437.log.gz 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23186.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29351.log 2024-08-21T23:34:39.038 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23186.log.gz 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115129.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74172.log 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115129.log.gz 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29351.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51661.log 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29351.log.gz 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74172.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40874.log 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74172.log.gz 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51661.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49998.log 2024-08-21T23:34:39.039 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51661.log.gz 2024-08-21T23:34:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40874.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105990.log 2024-08-21T23:34:39.040 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40874.log.gz 2024-08-21T23:34:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49998.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.49998.log.gz 2024-08-21T23:34:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.84506.log 2024-08-21T23:34:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105990.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60151.log 2024-08-21T23:34:39.040 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105990.log.gz 2024-08-21T23:34:39.040 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84506.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117069.log 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84506.log.gz 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60151.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111529.log 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60151.log.gz 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117069.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.117069.log.gz 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.99890.log 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111529.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102014.log 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111529.log.gz 2024-08-21T23:34:39.041 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99890.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36539.log 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99890.log.gz 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102014.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89177.log 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102014.log.gz 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36539.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42163.log 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36539.log.gz 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89177.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119829.log 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89177.log.gz 2024-08-21T23:34:39.042 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42163.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59229.log 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42163.log.gz 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119829.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38201.log 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119829.log.gz 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59229.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116517.log 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59229.log.gz 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38201.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.65407.log 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38201.log.gz 2024-08-21T23:34:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116517.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.30920.log 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116517.log.gz 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65407.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.65407.log.gz -5 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/c1-client.0.92133.log 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30920.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91486.log 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30920.log.gz 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92133.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111345.log 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92133.log.gz 2024-08-21T23:34:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91486.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44560.log 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91486.log.gz 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111345.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.111345.log.gz 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.72047.log 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.99522.log 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44560.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.44560.log.gz 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72047.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76662.log 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72047.log.gz 2024-08-21T23:34:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99522.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47141.log 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99522.log.gz 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76662.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50460.log 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76662.log.gz 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47141.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107283.log 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47141.log.gz 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50460.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72787.log 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50460.log.gz 2024-08-21T23:34:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107283.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72233.log 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107283.log.gz 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72787.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77769.log 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72787.log.gz 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72233.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101554.log 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72233.log.gz 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77769.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118446.log 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77769.log.gz 2024-08-21T23:34:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101554.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61533.log 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101554.log.gz 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118446.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68451.log 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118446.log.gz 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61533.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34054.log 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61533.log.gz 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68451.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119737.log 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68451.log.gz 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/c1-client.0.34054.log: --verbose -- /var/log/ceph/c1-client.0.55719.log 2024-08-21T23:34:39.048 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34054.log.gz 2024-08-21T23:34:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119737.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114576.log 2024-08-21T23:34:39.049 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119737.log.gz 2024-08-21T23:34:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55719.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115592.log 2024-08-21T23:34:39.049 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55719.log.gz 2024-08-21T23:34:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114576.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93055.log 2024-08-21T23:34:39.049 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114576.log.gz 2024-08-21T23:34:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115592.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71955.log 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115592.log.gz 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93055.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76570.log 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93055.log.gz 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71955.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38109.log 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71955.log.gz 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118814.log 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76570.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.76570.log.gz 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38109.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51291.log 2024-08-21T23:34:39.050 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38109.log.gz 2024-08-21T23:34:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118814.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92503.log 2024-08-21T23:34:39.051 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118814.log.gz 2024-08-21T23:34:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51291.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73618.log 2024-08-21T23:34:39.051 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51291.log.gz 2024-08-21T23:34:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92503.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.92503.log.gz 2024-08-21T23:34:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.112270.log 2024-08-21T23:34:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73618.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99614.log 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73618.log.gz 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112270.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75371.log 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112270.log.gz 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99614.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75649.log 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99614.log.gz 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75371.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79429.log 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75371.log.gz 2024-08-21T23:34:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75649.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110793.log 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75649.log.gz 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79429.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44100.log 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79429.log.gz 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110793.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56826.log 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110793.log.gz 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44100.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.44100.log.gz 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.78138.log 2024-08-21T23:34:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56826.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120199.log 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56826.log.gz 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78138.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61257.log 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78138.log.gz 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120199.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75095.log 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120199.log.gz 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61257.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31927.log 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61257.log.gz 2024-08-21T23:34:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75095.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46679.log 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75095.log.gz 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31927.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81920.log 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31927.log.gz 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46679.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.46679.log.gz 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.117625.log 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/c1-client.0.81920.log: -5 --verbose -- /var/log/ceph/c1-client.0.74080.log 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81920.log.gz 2024-08-21T23:34:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117625.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50923.log 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117625.log.gz 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74080.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74356.log 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74080.log.gz 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50923.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52586.log 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50923.log.gz 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74356.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92225.log 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74356.log.gz 2024-08-21T23:34:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52586.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31746.log 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52586.log.gz 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92225.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.92225.log.gzgzip 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.71771.log 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31746.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66790.log 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31746.log.gz 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71771.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45665.log 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71771.log.gz 2024-08-21T23:34:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66790.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63564.log 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66790.log.gz 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45665.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88008.log 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45665.log.gz 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63564.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57659.log 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63564.log.gz 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88008.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.27133.log 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88008.log.gz 2024-08-21T23:34:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57659.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34882.log 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57659.log.gz 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.27133.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110146.log 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.27133.log.gz 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34882.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102290.log 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34882.log.gz 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110146.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73342.log 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110146.log.gz 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102290.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89361.log 2024-08-21T23:34:39.059 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102290.log.gz 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73342.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52308.log 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73342.log.gz 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89361.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77677.log 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89361.log.gz 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52308.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43451.log 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52308.log.gz 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77677.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99706.log 2024-08-21T23:34:39.060 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77677.log.gz 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.45851.log 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43451.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.43451.log.gz 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99706.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56550.log 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99706.log.gz 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45851.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32292.log 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45851.log.gz 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56550.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83118.log 2024-08-21T23:34:39.061 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56550.log.gz 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32292.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32764.log 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32292.log.gz 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83118.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89732.log 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83118.log.gz 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32764.log: gzip -5 --verbose 0.0% -- /var/log/ceph/c1-client.0.106358.log -- replaced with /var/log/ceph/c1-client.0.32764.log.gz 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.89732.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52030.log 2024-08-21T23:34:39.062 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.89732.log.gz 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106358.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85150.log 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106358.log.gz 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52030.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115869.log 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52030.log.gz 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85150.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94348.log 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85150.log.gz 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115869.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98229.log 2024-08-21T23:34:39.063 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115869.log.gz 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94348.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96109.log 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94348.log.gz 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98229.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.24036.log 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98229.log.gz 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.46587.log 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96109.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.96109.log.gz 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.24036.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77953.log 2024-08-21T23:34:39.064 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.24036.log.gz 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46587.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48616.log 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46587.log.gz 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77953.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64854.log 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77953.log.gz 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48616.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37459.log 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48616.log.gz 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64854.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49538.log 2024-08-21T23:34:39.065 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64854.log.gz 2024-08-21T23:34:39.066 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37459.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69742.log 2024-08-21T23:34:39.066 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37459.log.gz 2024-08-21T23:34:39.066 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49538.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38753.log 2024-08-21T23:34:39.066 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49538.log.gz 2024-08-21T23:34:39.066 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69742.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.28891.log 2024-08-21T23:34:39.066 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69742.log.gz 2024-08-21T23:34:39.066 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38753.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62642.log 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38753.log.gz 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.114208.log 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.28891.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.28891.log.gz 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62642.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87087.log 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62642.log.gz 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114208.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.114208.log.gz 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.88193.log 2024-08-21T23:34:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87087.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95370.log 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87087.log.gz 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88193.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75557.log 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88193.log.gz 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95370.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53694.log 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95370.log.gz 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75557.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62271.log 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75557.log.gz 2024-08-21T23:34:39.068 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53694.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.90378.log 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.53694.log.gz 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62271.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67159.log 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62271.log.gz 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90378.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92411.log 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90378.log.gz 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67159.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43732.log 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67159.log.gz 2024-08-21T23:34:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92411.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82839.log 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92411.log.gz 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43732.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113008.log 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43732.log.gz 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82839.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71679.log 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82839.log.gz 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113008.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94071.log 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113008.log.gz 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71679.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85704.log 2024-08-21T23:34:39.070 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71679.log.gz 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94071.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/c1-client.0.94071.log.gz /var/log/ceph/c1-client.admin.23308.log 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85704.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94713.log 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85704.log.gz 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23308.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63287.log 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23308.log.gz 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94713.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51938.log 2024-08-21T23:34:39.071 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94713.log.gz 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63287.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.63287.log.gz 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.115223.log 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51938.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101278.log 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51938.log.gz 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115223.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54248.log 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115223.log.gz 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101278.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103398.log 2024-08-21T23:34:39.072 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101278.log.gz 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54248.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.37183.log 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.54248.log.gz 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103398.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79892.log 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103398.log.gz 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37183.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90563.log 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37183.log.gz 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79892.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71497.log 2024-08-21T23:34:39.073 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79892.log.gz 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90563.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.51477.log 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90563.log.gz 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71497.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120107.log 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71497.log.gz 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.51477.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56366.log 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.51477.log.gz 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120107.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91949.log 2024-08-21T23:34:39.074 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120107.log.gz 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56366.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121585.log 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56366.log.gz 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91949.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95462.log 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91949.log.gz 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121585.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76293.log 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121585.log.gz 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95462.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99244.log 2024-08-21T23:34:39.075 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95462.log.gz 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76293.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33041.log 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76293.log.gz 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99244.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57199.log 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99244.log.gz 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33041.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53417.log 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.33041.log.gz 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57199.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31476.log 2024-08-21T23:34:39.076 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57199.log.gz 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53417.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42624.log 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53417.log.gz 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31476.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36354.log 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31476.log.gz 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42624.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91762.log 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42624.log.gz 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36354.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91670.log 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36354.log.gz 2024-08-21T23:34:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91762.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72695.log 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91762.log.gz 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91670.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55535.log 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91670.log.gz 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72695.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61903.log 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72695.log.gz 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55535.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.55535.log.gz 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.42900.log 2024-08-21T23:34:39.078 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61903.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74264.log 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61903.log.gz 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42900.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.42900.log.gz 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.38385.log 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74264.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96661.log 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74264.log.gz 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38385.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.62919.log 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38385.log.gz 2024-08-21T23:34:39.079 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.103492.log 2024-08-21T23:34:39.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96661.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.96661.log.gz 2024-08-21T23:34:39.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62919.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64300.log 2024-08-21T23:34:39.080 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62919.log.gz 2024-08-21T23:34:39.080 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103492.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.34974.log 2024-08-21T23:34:39.080 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103492.log.gz 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64300.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106820.log 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64300.log.gz 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34974.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109219.log 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34974.log.gz 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106820.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96753.log 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106820.log.gz 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109219.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52956.log 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109219.log.gz 2024-08-21T23:34:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96753.log: gzip -5 --verbose -- /var/log/ceph/c1-mon.a.log 2024-08-21T23:34:39.082 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96753.log.gz 2024-08-21T23:34:39.082 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52956.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36999.log 2024-08-21T23:34:39.082 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52956.log.gz 2024-08-21T23:34:39.082 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.25856.log 2024-08-21T23:34:39.082 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36999.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56458.log 2024-08-21T23:34:39.082 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36999.log.gz 2024-08-21T23:34:39.082 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.25856.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64392.log 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.25856.log.gz 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56458.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49814.log 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56458.log.gz 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64392.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.113192.log 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64392.log.gz 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49814.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111161.log 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49814.log.gz 2024-08-21T23:34:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.113192.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112362.log 2024-08-21T23:34:39.084 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.113192.log.gz 2024-08-21T23:34:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111161.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54890.log 2024-08-21T23:34:39.084 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111161.log.gz 2024-08-21T23:34:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112362.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.30547.log 2024-08-21T23:34:39.084 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112362.log.gz 2024-08-21T23:34:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54890.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76939.log 2024-08-21T23:34:39.084 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54890.log.gz 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30547.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116609.log 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30547.log.gz 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76939.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106174.log 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76939.log.gz 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116609.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99152.log 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116609.log.gz 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106174.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103122.log 2024-08-21T23:34:39.085 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106174.log.gz 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.52772.log 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99152.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.99152.log.gz 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103122.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.103122.log.gz 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.114760.log 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52772.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42808.log 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52772.log.gz 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114760.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121769.log 2024-08-21T23:34:39.086 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114760.log.gz 2024-08-21T23:34:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50090.log 2024-08-21T23:34:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42808.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.42808.log.gz 2024-08-21T23:34:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121769.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33225.log 2024-08-21T23:34:39.087 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121769.log.gz 2024-08-21T23:34:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50090.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.50090.log.gz 2024-08-21T23:34:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.70297.log 2024-08-21T23:34:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33225.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60610.log 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.33225.log.gz 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70297.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.70297.log.gz 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.19546.log 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60610.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90286.log 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60610.log.gz 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.19546.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35342.log 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.19546.log.gz 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90286.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40137.log 2024-08-21T23:34:39.088 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90286.log.gz 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35342.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44192.log 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35342.log.gz 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40137.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119277.log 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40137.log.gz 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44192.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42532.log 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44192.log.gz 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119277.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.119277.log.gz 2024-08-21T23:34:39.089 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.73155.log 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42532.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99798.log 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42532.log.gz 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73155.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71220.log 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73155.log.gz 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99798.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79337.log 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99798.log.gz 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71220.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.71220.log.gz 2024-08-21T23:34:39.090 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.66974.log 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79337.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82656.log 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79337.log.gz 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66974.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.66974.log.gz 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.97216.log 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82656.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29811.log 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82656.log.gz 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97216.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c1-client.0.97216.log.gz --verbose 2024-08-21T23:34:39.091 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/c1-client.0.105805.log 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29811.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70113.log 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29811.log.gz 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105805.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.105805.log.gz 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80077.log 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70113.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107099.log 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70113.log.gz 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80077.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112824.log 2024-08-21T23:34:39.092 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.80077.log.gz 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107099.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88901.log 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107099.log.gz 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.101370.log 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112824.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.112824.log.gz 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88901.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81643.log 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88901.log.gz 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101370.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49354.log 2024-08-21T23:34:39.093 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.101370.log.gz 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.104881.log 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81643.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.81643.log.gz 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49354.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23436.log 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49354.log.gz 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104881.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47511.log 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104881.log.gz 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23436.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94163.log 2024-08-21T23:34:39.094 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23436.log.gz 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47511.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58027.log 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47511.log.gz 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94163.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58767.log 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94163.log.gz 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58027.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66514.log 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58027.log.gz 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58767.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61717.log 2024-08-21T23:34:39.095 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58767.log.gz 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66514.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67066.log 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66514.log.gz 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61717.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.120568.log 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61717.log.gz 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67066.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57383.log 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67066.log.gz 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.120568.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55903.log 2024-08-21T23:34:39.096 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.120568.log.gz 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57383.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121028.log 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57383.log.gz 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55903.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70205.log 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55903.log.gz 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121028.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/c1-client.0.121028.log.gz -- 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/c1-client.0.28983.log 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/c1-client.0.70205.log: -- /var/log/ceph/c1-client.0.53141.log 2024-08-21T23:34:39.097 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70205.log.gz 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.28983.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48064.log 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.28983.log.gz 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53141.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56182.log 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53141.log.gz 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.50736.log 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48064.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.48064.log.gz 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56182.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87455.log 2024-08-21T23:34:39.098 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56182.log.gz 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50736.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84782.log 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50736.log.gz 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87455.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105529.log 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87455.log.gz 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84782.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48800.log 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84782.log.gz 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105529.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.121307.log 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.105529.log.gz 2024-08-21T23:34:39.099 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48800.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100653.log 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48800.log.gz 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.121307.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97493.log 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.121307.log.gz 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.100653.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.86719.log 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.100653.log.gz 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97493.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93239.log 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97493.log.gz 2024-08-21T23:34:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86719.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72326.log 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86719.log.gz 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93239.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.63380.log 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93239.log.gz 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.97030.log 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72326.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.72326.log.gz 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.63380.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54340.log 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.63380.log.gz 2024-08-21T23:34:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97030.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112730.log 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97030.log.gz 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.98324.log 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54340.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.54340.log.gz 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112730.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.69466.log 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112730.log.gz 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.96293.log 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98324.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.98324.log.gz 2024-08-21T23:34:39.102 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69466.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.81275.log 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69466.log.gz 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96293.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108206.log 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96293.log.gz 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81275.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95002.log 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81275.log.gz 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108206.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39582.log 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108206.log.gz 2024-08-21T23:34:39.103 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95002.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.95002.log.gz 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.102660.log 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.67527.log 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39582.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39582.log.gz 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102660.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29259.log 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102660.log.gz 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67527.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44929.log 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67527.log.gz 2024-08-21T23:34:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29259.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102474.log 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29259.log.gz 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44929.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.44929.log.gz 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62458.log 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102474.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.22928.log 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102474.log.gz 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62458.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.62458.log.gz 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.109314.log 2024-08-21T23:34:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.22928.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92041.log 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.22928.log.gz 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.93147.log 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109314.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.109314.log.gz 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92041.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58859.log 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92041.log.gz 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93147.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78322.log 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93147.log.gz 2024-08-21T23:34:39.106 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58859.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93795.log 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58859.log.gz 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78322.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108483.log 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.78322.log.gz 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93795.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.93795.log.gz 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.34330.log 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108483.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.102568.log 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.108483.log.gz 2024-08-21T23:34:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34330.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99982.log 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34330.log.gz 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102568.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.102568.log.gz 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69190.log 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99982.log: gzip -5 --verbose -- /var/log/ceph/c1-client.1.24358.log 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99982.log.gz 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69190.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64668.log 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69190.log.gz 2024-08-21T23:34:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.1.24358.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109406.log 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64668.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44284.log 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64668.log.gz 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109406.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.1.log 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109406.log.gz 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44284.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.60425.log 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44284.log.gz 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.54706.log 2024-08-21T23:34:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/rgw.c1.client.1.log: /var/log/ceph/c1-client.0.60425.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79521.log 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60425.log.gz 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54706.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82750.log 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54706.log.gz 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79521.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84414.log 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79521.log.gz 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82750.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103306.log 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82750.log.gz 2024-08-21T23:34:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84414.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.84414.log.gz 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.34514.log 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103306.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78690.log 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103306.log.gz 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.34514.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55351.log 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.34514.log.gz 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr: 84.6% -- replaced with /var/log/ceph/c1-client.1.24358.log.gzgzip 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.32019.log 2024-08-21T23:34:39.111 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78690.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.78690.log.gz 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55351.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.55351.log.gz 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.admin.123312.log 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32019.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.79708.log 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32019.log.gz 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.123312.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23690.log 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.123312.log.gz 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.79708.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42347.log 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.79708.log.gz 2024-08-21T23:34:39.112 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23690.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111808.log 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23690.log.gz 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42347.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82564.log 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42347.log.gz 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111808.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110517.log 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111808.log.gz 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82564.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.47049.log 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82564.log.gz 2024-08-21T23:34:39.113 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110517.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97309.log 2024-08-21T23:34:39.114 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110517.log.gz 2024-08-21T23:34:39.114 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.60332.log 2024-08-21T23:34:39.114 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47049.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.47049.log.gz 2024-08-21T23:34:39.114 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97309.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23732.log 2024-08-21T23:34:39.114 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97309.log.gz 2024-08-21T23:34:39.114 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.60332.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85980.log 2024-08-21T23:34:39.114 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.60332.log.gz 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23732.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.95831.log 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23732.log.gz 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85980.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.30823.log 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85980.log.gz 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.95831.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.103214.log 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.95831.log.gz 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30823.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55075.log 2024-08-21T23:34:39.115 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30823.log.gz 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.103214.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.66422.log 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.103214.log.gz 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55075.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.55075.log.gz 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56274.log 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.66422.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58399.log 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.66422.log.gz 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56274.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57104.log 2024-08-21T23:34:39.116 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56274.log.gz 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58399.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115037.log 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58399.log.gz 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57104.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.59321.log 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57104.log.gz 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115037.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.118262.log 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115037.log.gz 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.59321.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75833.log 2024-08-21T23:34:39.117 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.59321.log.gz 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.122322.log 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118262.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.118262.log.gz 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75833.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75463.log 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75833.log.gz 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.122322.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32856.log 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122322.log.gz 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75463.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44745.log 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75463.log.gz 2024-08-21T23:34:39.118 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32856.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.108943.log 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32856.log.gz 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44745.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53602.log 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44745.log.gz 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.108943.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.108943.log.gz 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.30639.log 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53602.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68822.log 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53602.log.gz 2024-08-21T23:34:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30639.log: gzip -5 --verbose -- /var/log/ceph/rgw.c1.client.0.log 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30639.log.gz 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68822.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.31838.log 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68822.log.gz 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/rgw.c1.client.0.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88459.log 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.31838.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54064.log 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.31838.log.gz 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88459.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.88459.log.gz 2024-08-21T23:34:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118909.log 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54064.log: gzip 0.0% -- replaced with /var/log/ceph/c1-client.0.54064.log.gz -5 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/c1-client.0.86533.log 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118909.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.61349.log 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118909.log.gz 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86533.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42440.log 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.86533.log.gz 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.61349.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.46495.log 2024-08-21T23:34:39.121 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.61349.log.gz 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.47787.log 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42440.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.42440.log.gz 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46495.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90102.log 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46495.log.gz 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.47787.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43916.log 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.47787.log.gz 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.119093.log 2024-08-21T23:34:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90102.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.90102.log.gz 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43916.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109127.log 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43916.log.gz 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119093.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58120.log 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119093.log.gz 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109127.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.67714.log 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109127.log.gz 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58120.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88101.log 2024-08-21T23:34:39.123 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58120.log.gz 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.67714.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.112084.log 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.67714.log.gz 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88101.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.101922.log 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88101.log.gz 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.112084.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.89269.log 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.85242.log 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.118722.log 2024-08-21T23:34:39.124 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.112084.log.gz 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.101922.log: /var/log/ceph/c1-client.0.89269.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.101922.log.gz 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85242.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.89269.log.gz 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.119461.log 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85242.log.gz 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.118722.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.39953.log 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.118722.log.gz 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119461.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98508.log 2024-08-21T23:34:39.125 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119461.log.gz 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.39953.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.39953.log.gz 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.36815.log 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98508.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92963.log 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98508.log.gz 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36815.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32111.log 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36815.log.gz 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92963.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45481.log 2024-08-21T23:34:39.126 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92963.log.gz 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32111.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45113.log 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.32111.log.gz 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45481.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.45481.log.gz 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.56918.log 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45113.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.19704.log 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.45113.log.gz 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56918.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110977.log 2024-08-21T23:34:39.127 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56918.log.gz 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.19704.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85519.log 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.19704.log.gz 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110977.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91394.log 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110977.log.gz 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.68359.log 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85519.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.85519.log.gz 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91394.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.75187.log 2024-08-21T23:34:39.128 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91394.log.gz 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68359.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.35986.log 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68359.log.gz 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.75187.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.72879.log 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.75187.log.gz 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.35986.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64760.log 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.35986.log.gz 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.72879.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94910.log 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.72879.log.gz 2024-08-21T23:34:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64760.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.42255.log 2024-08-21T23:34:39.130 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64760.log.gz 2024-08-21T23:34:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94910.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115961.log 2024-08-21T23:34:39.130 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94910.log.gz 2024-08-21T23:34:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42255.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48340.log 2024-08-21T23:34:39.130 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42255.log.gz 2024-08-21T23:34:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.62087.log 2024-08-21T23:34:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115961.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.115961.log.gz 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48340.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.119553.log 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48340.log.gz 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.62087.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.29719.log 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.62087.log.gz 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119553.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.54524.log -- replaced with /var/log/ceph/c1-client.0.119553.log.gz 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29719.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.78414.log 2024-08-21T23:34:39.131 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.29719.log.gz 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54524.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.80539.log 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54524.log.gz 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.78414.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/c1-client.0.78414.log.gz -- 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/c1-client.0.92595.log 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80539.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.80539.log.gz 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.37553.log 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92595.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54616.log 2024-08-21T23:34:39.132 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92595.log.gz 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37553.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26161.log 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37553.log.gz 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54616.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49076.log 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54616.log.gz 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.26161.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.32672.log 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.26161.log.gz 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49076.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73434.log 2024-08-21T23:34:39.133 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49076.log.gz 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.65223.log 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.32672.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.32672.log.gz 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73434.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45297.log 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.73434.log.gz 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.65223.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36723.log 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.65223.log.gz 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.81459.log 2024-08-21T23:34:39.134 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45297.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.45297.log.gz 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36723.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116885.log 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36723.log.gz 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.81459.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.84598.log 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.81459.log.gz 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55167.log 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116885.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.116885.log.gz 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.84598.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.91117.log 2024-08-21T23:34:39.135 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.84598.log.gz 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55167.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96477.log 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55167.log.gz 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91117.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71312.log 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91117.log.gz 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96477.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110701.log 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96477.log.gz 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71312.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58491.log 2024-08-21T23:34:39.136 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.71312.log.gz 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110701.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40230.log 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.110701.log.gz 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.102382.log 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58491.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.58491.log.gz 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40230.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.115500.log 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40230.log.gz 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.102382.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92318.log 2024-08-21T23:34:39.137 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.102382.log.gz 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.115500.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117813.log 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.115500.log.gz 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92318.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.122414.log 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92318.log.gz 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117813.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.97122.log 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117813.log.gz 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.122414.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58212.log 2024-08-21T23:34:39.138 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.122414.log.gz 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.97122.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.44376.log 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.97122.log.gz 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.80446.log 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58212.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.58212.log.gz 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.44376.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.111992.log 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.44376.log.gz 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.80446.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.80446.log.gz 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.46311.log 2024-08-21T23:34:39.139 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111992.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23144.log 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111992.log.gz 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.46311.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.41979.log 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.46311.log.gz 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23144.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.64484.log 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23144.log.gz 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.41979.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.41979.log.gz 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.83492.log 2024-08-21T23:34:39.140 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.64484.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.49262.log 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.64484.log.gz 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83492.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.37275.log 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83492.log.gz 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49262.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57935.log 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49262.log.gz 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37275.log: gzip -5 --verbose 0.0% -- /var/log/ceph/c1-client.0.91578.log -- replaced with /var/log/ceph/c1-client.0.37275.log.gz 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:39.141 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57935.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110238.log 2024-08-21T23:34:39.142 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57935.log.gz 2024-08-21T23:34:39.142 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.91578.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.94818.log 2024-08-21T23:34:39.142 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.91578.log.gz 2024-08-21T23:34:39.142 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110238.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/c1-client.0.110238.log.gz /var/log/ceph/c1-client.0.55443.log 2024-08-21T23:34:39.142 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:34:39.142 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94818.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106912.log 2024-08-21T23:34:39.142 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94818.log.gz 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55443.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.93609.log 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55443.log.gz 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106912.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.83676.log 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106912.log.gz 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93609.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57751.log 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93609.log.gz 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83676.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90010.log 2024-08-21T23:34:39.143 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83676.log.gz 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57751.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98876.log 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57751.log.gz 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90010.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.24276.log 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.90010.log.gz 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98876.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70942.log 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.98876.log.gz 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.24276.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.68914.log 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70942.log: gzip -5 --verbose -- /var/log/ceph/c1.log 2024-08-21T23:34:39.144 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70942.log.gz 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.68914.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43824.log 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.68914.log.gz 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.85796.log 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43824.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.43824.log.gz 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.106266.log 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.85796.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.30271.log 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.85796.log.gz 2024-08-21T23:34:39.145 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106266.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.107007.log 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106266.log.gz 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.30271.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.100240.log 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.30271.log.gz 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr: 76.9% -- replaced with /var/log/ceph/c1-client.0.24276.log.gz 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107007.log: gzip -5 --verbose -- 0.0% /var/log/ceph/c1-client.0.49630.log 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/c1-client.0.107007.log.gz 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.100240.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.73063.log 2024-08-21T23:34:39.146 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.49630.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.56090.log 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.49630.log.gz 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.116701.log 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.73063.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.73063.log.gz 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.56090.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.71587.log 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.56090.log.gz 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116701.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.96385.log 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr: 66.1% -- replaced with /var/log/ceph/c1-client.0.100240.log.gz 2024-08-21T23:34:39.147 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116701.log.gz 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.40506.log 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.71587.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.71587.log.gz 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96385.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.52400.log 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.96385.log.gz 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40506.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.40506.log.gz 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.55259.log 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.52400.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.116239.log 2024-08-21T23:34:39.148 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.52400.log.gz 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55259.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43084.log 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55259.log.gz 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.116239.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.92779.log 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.116239.log.gz 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43084.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105344.log 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43084.log.gz 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.92779.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.23954.log 2024-08-21T23:34:39.149 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.92779.log.gz 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105344.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/c1-client.0.105344.log.gz -- 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/c1-client.0.94529.log 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.23954.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53786.log 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.23954.log.gz 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.94529.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.90655.log 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.94529.log.gz 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53786.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82380.log 2024-08-21T23:34:39.150 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53786.log.gz 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.107559.log 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.90655.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.90655.log.gz 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82380.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70573.log 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82380.log.gz 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.107559.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104140.log 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.107559.log.gz 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70573.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.33133.log 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.70573.log.gz 2024-08-21T23:34:39.151 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104140.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.74450.log 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104140.log.gz 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.33133.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.87639.log 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.33133.log.gz 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74450.log: 0.0%gzip -- replaced with /var/log/ceph/c1-client.0.74450.log.gz 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/c1-client.0.111716.log 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.87639.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.88285.log 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.87639.log.gz 2024-08-21T23:34:39.152 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.111716.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.109683.log 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.111716.log.gz 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.88285.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.110054.log 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.88285.log.gz 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.109683.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36907.log 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.109683.log.gz 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.57843.log 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.110054.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.110054.log.gz 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36907.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.105065.log 2024-08-21T23:34:39.153 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36907.log.gz 2024-08-21T23:34:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57843.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.57843.log.gz 2024-08-21T23:34:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.74542.log 2024-08-21T23:34:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.37830.log 2024-08-21T23:34:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.105065.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.105065.log.gz 2024-08-21T23:34:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.74542.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.117253.log 2024-08-21T23:34:39.154 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.74542.log.gz 2024-08-21T23:34:39.154 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.37830.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38477.log 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.37830.log.gz 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.117253.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.99336.log 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.117253.log.gz 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38477.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.26119.log 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38477.log.gz 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.99336.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.98416.log 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.99336.log.gz 2024-08-21T23:34:39.155 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.26119.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c1-client.admin.26119.log.gz --verbose 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/c1-client.0.93979.log 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.86903.log 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.98416.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.98416.log.gz 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.93979.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.106082.log 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.93979.log.gz 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.86903.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/c1-client.0.86903.log.gz 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/c1-client.0.96569.log 2024-08-21T23:34:39.156 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.106082.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.27297.log 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.106082.log.gz 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.96569.log: gzip 0.0% -5 -- replaced with /var/log/ceph/c1-client.0.96569.log.gz --verbose 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/c1-client.0.83952.log 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.27297.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57475.log 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.27297.log.gz 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.83952.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.27379.log 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.83952.log.gz 2024-08-21T23:34:39.157 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/c1-client.0.57475.log: -5 --verbose -- /var/log/ceph/c1-client.0.42716.log 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.57475.log.gz 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.27379.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.58951.log 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.42716.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.43546.log 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.42716.log.gz 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.58951.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.77584.log 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.58951.log.gz 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.43546.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.40414.log 2024-08-21T23:34:39.158 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.43546.log.gz 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.77584.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.38845.log 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.77584.log.gz 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.40414.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55811.log 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.40414.log.gz 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.38845.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.36262.log 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.38845.log.gz 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55811.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.114668.log 2024-08-21T23:34:39.159 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55811.log.gz 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.36262.log: gzip -5 --verbose -- /var/log/ceph/c1-client.admin.19883.log 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.36262.log.gz 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.114668.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.53325.log 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.114668.log.gz 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/c1.log.gz 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.admin.19883.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48248.log 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.admin.19883.log.gz 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.53325.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.104232.log 2024-08-21T23:34:39.160 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.53325.log.gz 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48248.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.48248.log.gz 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.119001.log 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr: 19.7% -- replaced with /var/log/ceph/c1-client.0.27379.log.gz 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.104232.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.48432.log 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.104232.log.gz 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.119001.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.55995.log 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.119001.log.gz 2024-08-21T23:34:39.161 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.48432.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.54156.log 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.48432.log.gz 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.55995.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.57567.log 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.55995.log.gz 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.54156.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.50644.log 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.54156.log.gz 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.69558.log 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.57567.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.57567.log.gz 2024-08-21T23:34:39.162 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.50644.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.76754.log 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.50644.log.gz 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.69558.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.82288.log 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.69558.log.gz 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.76754.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.45205.log 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.76754.log.gz 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.82288.log: gzip -5 --verbose -- /var/log/ceph/c1-client.0.70021.log 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/c1-client.0.82288.log.gz 2024-08-21T23:34:39.163 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.45205.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.45205.log.gz 2024-08-21T23:34:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/c1-client.0.29167.log 2024-08-21T23:34:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.70021.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.70021.log.gz 2024-08-21T23:34:39.164 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/c1-client.0.29167.log: 0.0% -- replaced with /var/log/ceph/c1-client.0.29167.log.gz 2024-08-21T23:34:41.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~125s 2024-08-21T23:34:41.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~125s 2024-08-21T23:34:41.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~125s 2024-08-21T23:34:41.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~125s 2024-08-21T23:34:41.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~125s 2024-08-21T23:34:41.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~125s 2024-08-21T23:34:41.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~125s 2024-08-21T23:34:46.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~130s 2024-08-21T23:34:46.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~130s 2024-08-21T23:34:46.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~130s 2024-08-21T23:34:46.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~130s 2024-08-21T23:34:46.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~130s 2024-08-21T23:34:46.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~130s 2024-08-21T23:34:46.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~130s 2024-08-21T23:34:51.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~135s 2024-08-21T23:34:51.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~135s 2024-08-21T23:34:51.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~135s 2024-08-21T23:34:51.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~135s 2024-08-21T23:34:51.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2024-08-21T23:34:51.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~135s 2024-08-21T23:34:51.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~135s 2024-08-21T23:34:51.947 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/c1-mgr.x.log.gz 2024-08-21T23:34:56.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~140s 2024-08-21T23:34:56.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~140s 2024-08-21T23:34:56.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~140s 2024-08-21T23:34:56.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~140s 2024-08-21T23:34:56.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2024-08-21T23:34:56.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~140s 2024-08-21T23:34:56.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~140s 2024-08-21T23:35:01.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~145s 2024-08-21T23:35:01.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~145s 2024-08-21T23:35:01.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~145s 2024-08-21T23:35:01.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~145s 2024-08-21T23:35:01.219 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~145s 2024-08-21T23:35:01.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~145s 2024-08-21T23:35:01.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~145s 2024-08-21T23:35:01.956 INFO:teuthology.orchestra.run.smithi178.stderr: 91.7% -- replaced with /var/log/ceph/c2-mgr.x.log.gz 2024-08-21T23:35:06.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~150s 2024-08-21T23:35:06.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~150s 2024-08-21T23:35:06.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~150s 2024-08-21T23:35:06.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~150s 2024-08-21T23:35:06.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~150s 2024-08-21T23:35:06.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~150s 2024-08-21T23:35:06.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~150s 2024-08-21T23:35:11.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~155s 2024-08-21T23:35:11.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~155s 2024-08-21T23:35:11.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~155s 2024-08-21T23:35:11.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~155s 2024-08-21T23:35:11.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~155s 2024-08-21T23:35:11.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~155s 2024-08-21T23:35:11.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~155s 2024-08-21T23:35:16.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~160s 2024-08-21T23:35:16.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~160s 2024-08-21T23:35:16.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~160s 2024-08-21T23:35:16.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~160s 2024-08-21T23:35:16.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~160s 2024-08-21T23:35:16.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~160s 2024-08-21T23:35:16.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~160s 2024-08-21T23:35:21.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~165s 2024-08-21T23:35:21.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~165s 2024-08-21T23:35:21.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~165s 2024-08-21T23:35:21.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~165s 2024-08-21T23:35:21.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~165s 2024-08-21T23:35:21.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~165s 2024-08-21T23:35:21.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~165s 2024-08-21T23:35:26.230 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~170s 2024-08-21T23:35:26.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~170s 2024-08-21T23:35:26.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~170s 2024-08-21T23:35:26.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~170s 2024-08-21T23:35:26.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~170s 2024-08-21T23:35:26.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~170s 2024-08-21T23:35:26.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~170s 2024-08-21T23:35:31.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~175s 2024-08-21T23:35:31.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~175s 2024-08-21T23:35:31.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~175s 2024-08-21T23:35:31.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~175s 2024-08-21T23:35:31.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~175s 2024-08-21T23:35:31.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~175s 2024-08-21T23:35:31.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~175s 2024-08-21T23:35:36.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~180s 2024-08-21T23:35:36.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~180s 2024-08-21T23:35:36.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~180s 2024-08-21T23:35:36.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~180s 2024-08-21T23:35:36.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~180s 2024-08-21T23:35:36.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~180s 2024-08-21T23:35:36.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~180s 2024-08-21T23:35:36.748 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/c1-mon.a.log.gz 2024-08-21T23:35:41.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~185s 2024-08-21T23:35:41.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~185s 2024-08-21T23:35:41.237 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~185s 2024-08-21T23:35:41.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~185s 2024-08-21T23:35:41.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2024-08-21T23:35:41.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~185s 2024-08-21T23:35:41.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~185s 2024-08-21T23:35:46.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~190s 2024-08-21T23:35:46.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~190s 2024-08-21T23:35:46.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~190s 2024-08-21T23:35:46.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~190s 2024-08-21T23:35:46.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~190s 2024-08-21T23:35:46.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~190s 2024-08-21T23:35:46.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~190s 2024-08-21T23:35:51.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~195s 2024-08-21T23:35:51.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~195s 2024-08-21T23:35:51.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~195s 2024-08-21T23:35:51.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~195s 2024-08-21T23:35:51.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~195s 2024-08-21T23:35:51.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~195s 2024-08-21T23:35:51.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~195s 2024-08-21T23:35:56.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~200s 2024-08-21T23:35:56.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~200s 2024-08-21T23:35:56.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~200s 2024-08-21T23:35:56.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~200s 2024-08-21T23:35:56.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~200s 2024-08-21T23:35:56.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~200s 2024-08-21T23:35:56.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~200s 2024-08-21T23:36:01.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~205s 2024-08-21T23:36:01.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~205s 2024-08-21T23:36:01.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~205s 2024-08-21T23:36:01.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~205s 2024-08-21T23:36:01.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~205s 2024-08-21T23:36:01.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~205s 2024-08-21T23:36:01.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~205s 2024-08-21T23:36:01.349 INFO:teuthology.orchestra.run.smithi178.stderr: 92.3% -- replaced with /var/log/ceph/c2-mon.a.log.gz 2024-08-21T23:36:06.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~210s 2024-08-21T23:36:06.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~210s 2024-08-21T23:36:06.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~210s 2024-08-21T23:36:06.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~210s 2024-08-21T23:36:06.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~210s 2024-08-21T23:36:06.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~210s 2024-08-21T23:36:06.250 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~210s 2024-08-21T23:36:11.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~215s 2024-08-21T23:36:11.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~215s 2024-08-21T23:36:11.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~215s 2024-08-21T23:36:11.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~215s 2024-08-21T23:36:11.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~215s 2024-08-21T23:36:11.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~215s 2024-08-21T23:36:11.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~215s 2024-08-21T23:36:12.061 INFO:teuthology.orchestra.run.smithi116.stderr: 94.0% -- replaced with /var/log/ceph/rgw.c1.client.1.log.gz 2024-08-21T23:36:16.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~220s 2024-08-21T23:36:16.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~220s 2024-08-21T23:36:16.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~220s 2024-08-21T23:36:16.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~220s 2024-08-21T23:36:16.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~220s 2024-08-21T23:36:16.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~220s 2024-08-21T23:36:16.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~220s 2024-08-21T23:36:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~225s 2024-08-21T23:36:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~225s 2024-08-21T23:36:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~225s 2024-08-21T23:36:21.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~225s 2024-08-21T23:36:21.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~225s 2024-08-21T23:36:21.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~225s 2024-08-21T23:36:21.257 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~225s 2024-08-21T23:36:26.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~230s 2024-08-21T23:36:26.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~230s 2024-08-21T23:36:26.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~230s 2024-08-21T23:36:26.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~230s 2024-08-21T23:36:26.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2024-08-21T23:36:26.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~230s 2024-08-21T23:36:26.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~230s 2024-08-21T23:36:31.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~235s 2024-08-21T23:36:31.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~235s 2024-08-21T23:36:31.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~235s 2024-08-21T23:36:31.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~235s 2024-08-21T23:36:31.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~235s 2024-08-21T23:36:31.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~235s 2024-08-21T23:36:31.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~235s 2024-08-21T23:36:36.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~240s 2024-08-21T23:36:36.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~240s 2024-08-21T23:36:36.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~240s 2024-08-21T23:36:36.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~240s 2024-08-21T23:36:36.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~240s 2024-08-21T23:36:36.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~240s 2024-08-21T23:36:36.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~240s 2024-08-21T23:36:41.264 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~245s 2024-08-21T23:36:41.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~245s 2024-08-21T23:36:41.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~245s 2024-08-21T23:36:41.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~245s 2024-08-21T23:36:41.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~245s 2024-08-21T23:36:41.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~245s 2024-08-21T23:36:41.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~245s 2024-08-21T23:36:46.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~250s 2024-08-21T23:36:46.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~250s 2024-08-21T23:36:46.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~250s 2024-08-21T23:36:46.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~250s 2024-08-21T23:36:46.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~250s 2024-08-21T23:36:46.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~250s 2024-08-21T23:36:46.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~250s 2024-08-21T23:36:51.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~255s 2024-08-21T23:36:51.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~255s 2024-08-21T23:36:51.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~255s 2024-08-21T23:36:51.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~255s 2024-08-21T23:36:51.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~255s 2024-08-21T23:36:51.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~255s 2024-08-21T23:36:51.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~255s 2024-08-21T23:36:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~260s 2024-08-21T23:36:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~260s 2024-08-21T23:36:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~260s 2024-08-21T23:36:56.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~260s 2024-08-21T23:36:56.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~260s 2024-08-21T23:36:56.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~260s 2024-08-21T23:36:56.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~260s 2024-08-21T23:37:01.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~265s 2024-08-21T23:37:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~265s 2024-08-21T23:37:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~265s 2024-08-21T23:37:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~265s 2024-08-21T23:37:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~265s 2024-08-21T23:37:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~265s 2024-08-21T23:37:01.275 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~265s 2024-08-21T23:37:06.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~270s 2024-08-21T23:37:06.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~270s 2024-08-21T23:37:06.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~270s 2024-08-21T23:37:06.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~270s 2024-08-21T23:37:06.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~270s 2024-08-21T23:37:06.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~270s 2024-08-21T23:37:06.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~270s 2024-08-21T23:37:11.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~275s 2024-08-21T23:37:11.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~275s 2024-08-21T23:37:11.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~275s 2024-08-21T23:37:11.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~275s 2024-08-21T23:37:11.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2024-08-21T23:37:11.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~275s 2024-08-21T23:37:11.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~275s 2024-08-21T23:37:16.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~280s 2024-08-21T23:37:16.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~280s 2024-08-21T23:37:16.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~280s 2024-08-21T23:37:16.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~280s 2024-08-21T23:37:16.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2024-08-21T23:37:16.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~280s 2024-08-21T23:37:16.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~280s 2024-08-21T23:37:21.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~285s 2024-08-21T23:37:21.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~285s 2024-08-21T23:37:21.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~285s 2024-08-21T23:37:21.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~285s 2024-08-21T23:37:21.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~285s 2024-08-21T23:37:21.286 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~285s 2024-08-21T23:37:21.286 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~285s 2024-08-21T23:37:26.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~290s 2024-08-21T23:37:26.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~290s 2024-08-21T23:37:26.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~290s 2024-08-21T23:37:26.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~290s 2024-08-21T23:37:26.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~290s 2024-08-21T23:37:26.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~290s 2024-08-21T23:37:26.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~290s 2024-08-21T23:37:31.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~295s 2024-08-21T23:37:31.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~295s 2024-08-21T23:37:31.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~295s 2024-08-21T23:37:31.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~295s 2024-08-21T23:37:31.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~295s 2024-08-21T23:37:31.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~295s 2024-08-21T23:37:31.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~295s 2024-08-21T23:37:36.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.0 is failed for ~300s 2024-08-21T23:37:36.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~300s 2024-08-21T23:37:36.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~300s 2024-08-21T23:37:36.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mon.a is failed for ~300s 2024-08-21T23:37:36.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~300s 2024-08-21T23:37:36.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~300s 2024-08-21T23:37:36.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.mgr.x is failed for ~300s 2024-08-21T23:37:36.292 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-08-21T23:39:38.423 INFO:teuthology.orchestra.run.smithi116.stderr: 94.0% -- replaced with /var/log/ceph/rgw.c1.client.0.log.gz 2024-08-21T23:39:50.618 INFO:teuthology.orchestra.run.smithi178.stderr: 94.2% -- replaced with /var/log/ceph/rgw.c2.client.1.log.gz 2024-08-21T23:40:39.237 INFO:teuthology.orchestra.run.smithi178.stderr: 94.3% -- replaced with /var/log/ceph/rgw.c2.client.0.log.gz 2024-08-21T23:54:30.216 INFO:teuthology.orchestra.run.smithi116.stderr: 93.8% -- replaced with /var/log/ceph/c1-osd.2.log.gz 2024-08-21T23:56:00.692 INFO:teuthology.orchestra.run.smithi116.stderr: 93.8% -- replaced with /var/log/ceph/c1-osd.1.log.gz 2024-08-21T23:56:32.817 INFO:teuthology.orchestra.run.smithi116.stderr: 93.8% -- replaced with /var/log/ceph/c1-osd.0.log.gz 2024-08-21T23:56:32.819 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-21T23:56:32.819 INFO:teuthology.orchestra.run.smithi116.stderr:real 21m54.336s 2024-08-21T23:56:32.819 INFO:teuthology.orchestra.run.smithi116.stderr:user 7m2.061s 2024-08-21T23:56:32.820 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m22.681s 2024-08-22T06:48:05.868 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-08-22T06:48:05.878 DEBUG:teuthology.task.console_log:Killing console logger for smithi116 2024-08-22T06:48:05.879 DEBUG:teuthology.task.console_log:Killing console logger for smithi178 2024-08-22T06:48:05.879 DEBUG:teuthology.exit:Finished running handlers